PyLearn Keras 06: 学習の再現性の保証とパラメータの最適化

今回は、モデルの最適化に関する基本的な方法を説明します。いつも通りバックエンドのチェックから。GPUを使う場合は、plaidmlが返って来ればokです。

In [3]:
# kerasのバックエンドをチェック
import plaidml.keras
import keras
print(keras.backend.backend())
plaidml.keras.backend

1. 前処理

今回は、CNNでMnistの数字を当てるという課題を行います。前処理などは、Keras03で行なっていますので、そちらを参照ください。 まず、Mnistのデータロードを行います。

In [5]:
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import RMSprop

(x_train, y_train), (x_test, y_test) = mnist.load_data()

# 各種サイズ取得
num_train = len(y_train) # 教師データ数
num_test = len(y_test) # テストデータ数
print("教師データサイズ: ", num_train)
print("テストデータサイズ: ", num_test)
size_0 = x_train.shape[1] # 横サイズ
size_1 = x_train.shape[2] # 縦サイズ
dim_input_vec = size_0 * size_1

# 規格化
# グレースケール画像の場合、0が黒、255が白となっている。
# ニューラルは値のレンジが0〜1が望ましいので、そのように規格化
x_train = x_train / 255
x_test = x_test / 255

# クラス分類問題なので、クラスラベルをカテゴリカル変数に変換
num_class = 10
y_train = keras.utils.to_categorical(y_train, num_class)
y_test = keras.utils.to_categorical(y_test, num_class)
教師データサイズ:  60000
テストデータサイズ:  10000

一応、画像の表示

In [7]:
import matplotlib.pyplot as plt
print("0枚目の画像に書かれている数字: ", y_train[0])
plt.imshow(x_train[0, :, :], cmap='gray')
plt.show()
0枚目の画像に書かれている数字:  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]

CNNのためのサイズ変換

In [8]:
import numpy as np

num_train = len(x_train[:, 0,0]) # 教師データの数
num_test = len(x_test[:, 0,0]) # テストデータの数
size_rows = len(x_train[0, :,0]) # 画像、行サイズ
size_cols = len(x_train[0, 0, :]) # 画像、列サイズ

print("x_trainのshape: ", np.shape(x_train))
x_train_cnn = np.reshape(x_train, [num_train, size_rows, size_cols, 1])
x_test_cnn = np.reshape(x_test, [num_test, size_rows, size_cols, 1])
print("x_train_cnnのshape: ", np.shape(x_train_cnn))
x_trainのshape:  (60000, 28, 28)
x_train_cnnのshape:  (60000, 28, 28, 1)

2. 学習の再現性

続いて、学習の再現性について説明します。ニューラルネットワーク系のモデルは、学習時に乱数に身を委ねる箇所がいくつもあります。したがって、同じ構造のモデル、同じ学習条件を指定しても、まったく同じモデルはほぼ確実にできません。しかし、過去のモデルを再現できないというのは、よくありません。

これを体感するために、まず、同じモデルなのに違う判定結果が出るコードを示します。はじめに、構築したいモデルの構造を、定義しておきます。

In [55]:
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D

def model_making(mod):
    
    # モデル構造の定義
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu', input_shape=(28, 28, 1)))
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu'))
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu'))
    mod.add(Dropout(0.25))
    mod.add(Flatten())
    mod.add(Dense(30, activation='relu'))
    mod.add(Dropout(0.5))
    mod.add(Dense(10, activation='softmax'))
    
    # コスト関数などの設定
    mod.compile(loss='categorical_crossentropy', optimizer=RMSprop(lr=10**-3), metrics=['accuracy'])
    # lr=10**-3 は学習係数
    
    return mod

続いて、まったく同じ構造のモデルを2つ定義します。

In [56]:
# 1つめ
m1 = Sequential()
m1 = model_making(m1)

# 2つめ
m2 = Sequential()
m2 = model_making(m2)

モデルが定義できたところで、お互いのモデルを学習させてみます(備考: validation_splitとは、教師データの「後ろ」から、指定された割合だけ、学習に使用しないvalidation dataに割り当てることを意味します。「後ろ」からですので、教師データの並び順が、クラスラベルで綺麗に並んでいると、ひどい目にあうので注意してください)。

In [57]:
batch_size = 129
epochs = 2 # 学習回数(適宜、大きくすること)

print("\n ### モデル1")
m1_lp = m1.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)

print("\n ### モデル2")
m2_lp = m2.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)
 ### モデル1
Train on 54000 samples, validate on 6000 samples
Epoch 1/2
54000/54000 [==============================] - 15s 286us/step - loss: 1.3957 - acc: 0.5115 - val_loss: 0.5773 - val_acc: 0.8472
Epoch 2/2
54000/54000 [==============================] - 7s 125us/step - loss: 0.8726 - acc: 0.7076 - val_loss: 0.4409 - val_acc: 0.8823

 ### モデル2
Train on 54000 samples, validate on 6000 samples
Epoch 1/2
54000/54000 [==============================] - 16s 297us/step - loss: 1.2072 - acc: 0.5959 - val_loss: 0.4659 - val_acc: 0.8865
Epoch 2/2
54000/54000 [==============================] - 8s 147us/step - loss: 0.7801 - acc: 0.7526 - val_loss: 0.3032 - val_acc: 0.9040

さて、accとval_acc(正答率)がm1とm2で異なることがわかります。同じデータの正答率が異なるということは、異なる判定結果が得られていることになります。すなわち、m1とm2は異なるモデルに仕上がったことを意味します。これを回避するには、

良い例:

  • 乱数シードを設定
  • m1について、Sequentialでモデルを定義
  • m1について、fitで学習開始
  • 乱数シードをm1の場合と同じものに設定
  • m2について、Sequentialでモデルを定義
  • m2について、fitで学習開始

という流れのコードを書く必要があります。この順番を厳密に守らないとNGなので、注意してください。例えば、

ダメな例:

  • m1について、Sequentialでモデルを定義
  • 乱数シードを設定
  • m1について、fitで学習開始
  • m2について、Sequentialでモデルを定義
  • 乱数シードをm1の場合と同じものに設定
  • m2について、fitで学習開始

とか

ダメな例:

  • 乱数シードを設定
  • m1について、Sequentialでモデルを定義
  • m1について、fitで学習開始
  • m2について、Sequentialでモデルを定義
  • m2について、fitで学習開始

とか

ダメな例:

  • 乱数シードを設定
  • m1について、Sequentialでモデルを定義
  • 乱数シードをm1の場合と同じものに設定
  • m2について、Sequentialでモデルを定義
  • m1について、fitで学習開始
  • m2について、fitで学習開始

とかは、(おそらく)すべてNGです。記載する位置がまあまあシビアなので、慣れるまではきちんと再現されているかチェックしないと危険です。

In [92]:
import os

batch_size = 129
epochs = 1 # 学習回数(適宜、大きくすること)
se = 5 # 乱数シード番号(任意の自然数、なんでもok)

print("\n ### モデル1")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
m1 = Sequential()
m1 = model_making(m1)
m1_lp = m1.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)


print("\n ### モデル2")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
m2 = Sequential()
m2 = model_making(m2)
m2_lp = m2.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)
 ### モデル1
Train on 54000 samples, validate on 6000 samples
Epoch 1/1
54000/54000 [==============================] - 22s 412us/step - loss: 1.2214 - acc: 0.5800 - val_loss: 0.5109 - val_acc: 0.8805

 ### モデル2
Train on 54000 samples, validate on 6000 samples
Epoch 1/1
54000/54000 [==============================] - 17s 324us/step - loss: 1.2214 - acc: 0.5800 - val_loss: 0.5109 - val_acc: 0.8805

モデル1, 2でまったく同じ結果になりました。すなわち、同じ判定構造を持つモデルということになります。時間がある人はepoch=1ではなく、100くらいでやってみても良いかもしれません。

3. 良い学習係数の探索

再現可能なモデルが作れるようになったところで、学習係数の最適化を行ってみます。学習係数は、1回のパラメータ更新に対する、変化の大きさを意味します。このためにまず、学習係数を自由に変えることのできる関数を用意します。

In [93]:
def mod_lr(mod, learning_rate):
    
    # モデル構造の定義
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu', input_shape=(28, 28, 1)))
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu'))
    mod.add(Conv2D(filters=1, kernel_size=(5, 5), activation='relu'))
    mod.add(Dropout(0.25))
    mod.add(Flatten())
    mod.add(Dense(30, activation='relu'))
    mod.add(Dropout(0.5))
    mod.add(Dense(10, activation='softmax'))
    
    # コスト関数などの設定
    mod.compile(loss='categorical_crossentropy', optimizer=RMSprop(lr=learning_rate), metrics=['accuracy'])
    # mod_lr関数の第二引数が、RMSpropの引数に入力される
    
    return mod

続いて、3つの学習係数を持つモデルを定義し、学習させます。

  • mm1: 学習係数が0.1
  • mm2: 学習係数が0.01
  • mm3: 学習係数が0.001
In [113]:
batch_size = 129
epochs = 10 # 学習回数(適宜、大きくすること)
se = 0 # 乱数シード番号(任意の自然数、なんでもok)

print("\n ### モデル1: 学習係数=10^-1")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
mm1 = Sequential()
mm1 = mod_lr(mm1, 10**-1)
mm1_lp = mm1.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)

print("\n ### モデル2: 学習係数=10^-2")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
mm2 = Sequential()
mm2 = mod_lr(mm2, 10**-2)
mm2_lp = mm2.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)

print("\n ### モデル3: 学習係数=10^-3")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
mm3 = Sequential()
mm3 = mod_lr(mm3, 10**-3)
mm3_lp = mm3.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)
 ### モデル1: 学習係数=10^-1
Train on 54000 samples, validate on 6000 samples
Epoch 1/10
54000/54000 [==============================] - 20s 369us/step - loss: 2.3081 - acc: 0.1048 - val_loss: 2.2964 - val_acc: 0.1053
Epoch 2/10
54000/54000 [==============================] - 9s 163us/step - loss: 2.3061 - acc: 0.1047 - val_loss: nan - val_acc: 0.0978
Epoch 3/10
54000/54000 [==============================] - 9s 161us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 4/10
54000/54000 [==============================] - 9s 172us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 5/10
54000/54000 [==============================] - 10s 184us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 6/10
54000/54000 [==============================] - 9s 169us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 7/10
54000/54000 [==============================] - 9s 173us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 8/10
54000/54000 [==============================] - 9s 161us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 9/10
54000/54000 [==============================] - 9s 168us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978
Epoch 10/10
54000/54000 [==============================] - 10s 183us/step - loss: nan - acc: 0.0988 - val_loss: nan - val_acc: 0.0978

 ### モデル2: 学習係数=10^-2
Train on 54000 samples, validate on 6000 samples
Epoch 1/10
54000/54000 [==============================] - 27s 506us/step - loss: 0.9435 - acc: 0.6952 - val_loss: 1.8265 - val_acc: 0.3343
Epoch 2/10
54000/54000 [==============================] - 10s 192us/step - loss: 0.7931 - acc: 0.7441 - val_loss: 0.5340 - val_acc: 0.8585
Epoch 3/10
54000/54000 [==============================] - 10s 193us/step - loss: 0.8218 - acc: 0.7549 - val_loss: 1.7070 - val_acc: 0.7322
Epoch 4/10
54000/54000 [==============================] - 9s 175us/step - loss: 1.1442 - acc: 0.7131 - val_loss: 1.3140 - val_acc: 0.7678
Epoch 5/10
54000/54000 [==============================] - 9s 169us/step - loss: 1.3100 - acc: 0.7183 - val_loss: 1.9130 - val_acc: 0.3343
Epoch 6/10
54000/54000 [==============================] - 10s 177us/step - loss: 1.1963 - acc: 0.6089 - val_loss: 5.6304 - val_acc: 0.1065
Epoch 7/10
54000/54000 [==============================] - 9s 164us/step - loss: 1.7528 - acc: 0.5799 - val_loss: 1.2117 - val_acc: 0.8222
Epoch 8/10
54000/54000 [==============================] - 8s 154us/step - loss: 1.2419 - acc: 0.7388 - val_loss: 1.5599 - val_acc: 0.7293
Epoch 9/10
54000/54000 [==============================] - 9s 168us/step - loss: 1.0878 - acc: 0.7209 - val_loss: 0.7666 - val_acc: 0.8343
Epoch 10/10
54000/54000 [==============================] - 11s 207us/step - loss: 0.9080 - acc: 0.7497 - val_loss: 2.7382 - val_acc: 0.6850

 ### モデル3: 学習係数=10^-3
Train on 54000 samples, validate on 6000 samples
Epoch 1/10
54000/54000 [==============================] - 33s 615us/step - loss: 1.4584 - acc: 0.4981 - val_loss: 0.6115 - val_acc: 0.8527
Epoch 2/10
54000/54000 [==============================] - 9s 166us/step - loss: 0.8982 - acc: 0.7039 - val_loss: 0.3623 - val_acc: 0.8850
Epoch 3/10
54000/54000 [==============================] - 10s 191us/step - loss: 0.8019 - acc: 0.7340 - val_loss: 0.3862 - val_acc: 0.8917
Epoch 4/10
54000/54000 [==============================] - 12s 214us/step - loss: 0.7743 - acc: 0.7433 - val_loss: 0.3178 - val_acc: 0.9010
Epoch 5/10
54000/54000 [==============================] - 9s 172us/step - loss: 0.7446 - acc: 0.7521 - val_loss: 0.3545 - val_acc: 0.9002
Epoch 6/10
54000/54000 [==============================] - 12s 231us/step - loss: 0.7317 - acc: 0.7582 - val_loss: 0.3491 - val_acc: 0.9010
Epoch 7/10
54000/54000 [==============================] - 15s 270us/step - loss: 0.7205 - acc: 0.7627 - val_loss: 0.3206 - val_acc: 0.9047
Epoch 8/10
54000/54000 [==============================] - 14s 255us/step - loss: 0.7021 - acc: 0.7682 - val_loss: 0.3467 - val_acc: 0.9070
Epoch 9/10
54000/54000 [==============================] - 10s 177us/step - loss: 0.6919 - acc: 0.7725 - val_loss: 0.2770 - val_acc: 0.9070
Epoch 10/10
54000/54000 [==============================] - 9s 170us/step - loss: 0.6836 - acc: 0.7745 - val_loss: 0.3125 - val_acc: 0.9048

学習が終わったところで、acc(正答率)の推移を見てみましょう。loss(今回のモデルではクロスエントロピー)も大事ですが、最終的な評価は正答率で行うことが多いので、accだけをみます。

In [115]:
plt.figure()
plt.plot(mm1_lp.epoch, np.array(mm1_lp.history['acc']),label='Train $10^{-1}$', c="b", ls="-.")
plt.plot(mm2_lp.epoch, np.array(mm2_lp.history['acc']),label='Train $10^{-2}$', c="r", ls="-.")
plt.plot(mm3_lp.epoch, np.array(mm3_lp.history['acc']),label='Train $10^{-3}$', c="g", ls="-.")
plt.title("Training data")
plt.xlabel('Epoch')
plt.ylabel('Acc')
plt.grid()
plt.legend()

plt.figure()
plt.plot(mm1_lp.epoch, np.array(mm1_lp.history['val_acc']),label='Valid. $10^{-1}$', c="b", ls="-")
plt.plot(mm2_lp.epoch, np.array(mm2_lp.history['val_acc']),label='Valid. $10^{-2}$', c="r", ls="-")
plt.plot(mm3_lp.epoch, np.array(mm3_lp.history['val_acc']),label='Valid. $10^{-3}$', c="g", ls="-")
plt.title("Validation data")
plt.xlabel('Epoch')
plt.ylabel('Acc')
plt.grid()
plt.legend()
Out[115]:
<matplotlib.legend.Legend at 0x1621a2f98>
  • mm1: 学習係数が0.1 -> 学習できていない。全然ダメ
  • mm2: 学習係数が0.01 -> 学習はできているかもしれないが、すごく不安定
  • mm3: 学習係数が0.001 -> ちょっとずつではあるものの、安定して性能が上昇していきそう

ということがわかりました。したがって、学習係数が0.001のモデルが良いことになります。もちろん、epochが10でいいわけがないですし、学習係数が0.0001や0.00001の方がいいかもしれません。本番の場合はもうちょっとepochを増やして、いろいろなパターンの学習係数を試すと良いでしょう。

4. 良いepochの探索

良い学習係数がわかったところで、次は、良いepochを考えてみます。epochは大きければ大きいほど、教師データの正答率は高くなる傾向にあるものの、ときとして過学習を引き起こし、validation dataの性能低下をもたらします。そのため、良いepochの探索もとても大事になります。

このために、良い学習係数の探索で構築したモデルを再現させた上で、良いepochの探索を行います。必ずしも再現させる必要はありませんが、良い学習係数が0.001であるという事実は、その乱数シードの中だけで成立することかもしれないため、まったく同じモデルで各種のパラメータを最適化していくことが大事になります。

そういうわけで、良いepochの探索を始めます。ここではepochを500回、少し多めにとってみます。結果を見るとき、epochが1〜10のとき、学習係数の探索のときと同じ値を示しているかについても、注意深く見るようにしてください。違うなら、どこかで失敗しています。

In [117]:
epochs = 200 # 学習回数(適宜、大きくすること)
se = 0 # 乱数シード番号(任意の自然数、なんでもok)

print("\n ### モデル3: 学習係数=10^-3")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
mm3 = Sequential()
mm3 = mod_lr(mm3, 10**-3)
mm3_lp = mm3.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)
 ### モデル3: 学習係数=10^-3
Train on 54000 samples, validate on 6000 samples
Epoch 1/200
54000/54000 [==============================] - 23s 422us/step - loss: 1.4584 - acc: 0.4981 - val_loss: 0.6115 - val_acc: 0.8527
Epoch 2/200
54000/54000 [==============================] - 11s 208us/step - loss: 0.8982 - acc: 0.7039 - val_loss: 0.3623 - val_acc: 0.8850
Epoch 3/200
54000/54000 [==============================] - 10s 177us/step - loss: 0.8019 - acc: 0.7340 - val_loss: 0.3862 - val_acc: 0.8917
Epoch 4/200
54000/54000 [==============================] - 8s 139us/step - loss: 0.7743 - acc: 0.7433 - val_loss: 0.3178 - val_acc: 0.9010
Epoch 5/200
54000/54000 [==============================] - 7s 123us/step - loss: 0.7446 - acc: 0.7521 - val_loss: 0.3545 - val_acc: 0.9002
Epoch 6/200
54000/54000 [==============================] - 10s 193us/step - loss: 0.7317 - acc: 0.7582 - val_loss: 0.3491 - val_acc: 0.9010
Epoch 7/200
54000/54000 [==============================] - 7s 126us/step - loss: 0.7205 - acc: 0.7627 - val_loss: 0.3206 - val_acc: 0.9047
Epoch 8/200
54000/54000 [==============================] - 6s 119us/step - loss: 0.7021 - acc: 0.7682 - val_loss: 0.3467 - val_acc: 0.9070
Epoch 9/200
54000/54000 [==============================] - 9s 170us/step - loss: 0.6919 - acc: 0.7725 - val_loss: 0.2770 - val_acc: 0.9070
Epoch 10/200
54000/54000 [==============================] - 11s 198us/step - loss: 0.6836 - acc: 0.7745 - val_loss: 0.3125 - val_acc: 0.9048
Epoch 11/200
54000/54000 [==============================] - 9s 163us/step - loss: 0.6761 - acc: 0.7756 - val_loss: 0.3086 - val_acc: 0.9123
Epoch 12/200
54000/54000 [==============================] - 9s 162us/step - loss: 0.6624 - acc: 0.7815 - val_loss: 0.2953 - val_acc: 0.9215
Epoch 13/200
54000/54000 [==============================] - 10s 181us/step - loss: 0.6492 - acc: 0.7868 - val_loss: 0.3041 - val_acc: 0.9108
Epoch 14/200
54000/54000 [==============================] - 10s 188us/step - loss: 0.6510 - acc: 0.7866 - val_loss: 0.2675 - val_acc: 0.9135
Epoch 15/200
54000/54000 [==============================] - 13s 233us/step - loss: 0.6474 - acc: 0.7859 - val_loss: 0.2924 - val_acc: 0.9192
Epoch 16/200
54000/54000 [==============================] - 14s 255us/step - loss: 0.6299 - acc: 0.7930 - val_loss: 0.2415 - val_acc: 0.9205
Epoch 17/200
54000/54000 [==============================] - 11s 211us/step - loss: 0.6287 - acc: 0.7954 - val_loss: 0.2836 - val_acc: 0.9192
Epoch 18/200
54000/54000 [==============================] - 12s 230us/step - loss: 0.6244 - acc: 0.7965 - val_loss: 0.2936 - val_acc: 0.9007
Epoch 19/200
54000/54000 [==============================] - 18s 328us/step - loss: 0.6592 - acc: 0.7862 - val_loss: 0.2687 - val_acc: 0.9252
Epoch 20/200
54000/54000 [==============================] - 14s 251us/step - loss: 0.6112 - acc: 0.7995 - val_loss: 0.2419 - val_acc: 0.9243
Epoch 21/200
54000/54000 [==============================] - 15s 282us/step - loss: 0.6086 - acc: 0.7994 - val_loss: 0.2767 - val_acc: 0.9248
Epoch 22/200
54000/54000 [==============================] - 13s 233us/step - loss: 0.6050 - acc: 0.8017 - val_loss: 0.2503 - val_acc: 0.9153
Epoch 23/200
54000/54000 [==============================] - 14s 256us/step - loss: 0.6099 - acc: 0.8008 - val_loss: 0.3330 - val_acc: 0.8925
Epoch 24/200
54000/54000 [==============================] - 11s 205us/step - loss: 0.6489 - acc: 0.7848 - val_loss: 0.2343 - val_acc: 0.9227
Epoch 25/200
54000/54000 [==============================] - 11s 209us/step - loss: 0.6063 - acc: 0.8022 - val_loss: 0.2814 - val_acc: 0.9245
Epoch 26/200
54000/54000 [==============================] - 11s 195us/step - loss: 0.6071 - acc: 0.8019 - val_loss: 0.2263 - val_acc: 0.9270
Epoch 27/200
54000/54000 [==============================] - 12s 230us/step - loss: 0.5928 - acc: 0.8056 - val_loss: 0.2634 - val_acc: 0.9283
Epoch 28/200
54000/54000 [==============================] - 16s 289us/step - loss: 0.5913 - acc: 0.8081 - val_loss: 0.2519 - val_acc: 0.9178
Epoch 29/200
54000/54000 [==============================] - 10s 185us/step - loss: 0.6016 - acc: 0.8042 - val_loss: 0.2881 - val_acc: 0.9158
Epoch 30/200
54000/54000 [==============================] - 12s 219us/step - loss: 0.5981 - acc: 0.8052 - val_loss: 0.2598 - val_acc: 0.9147
Epoch 31/200
54000/54000 [==============================] - 11s 211us/step - loss: 0.6049 - acc: 0.8041 - val_loss: 0.2665 - val_acc: 0.9238
Epoch 32/200
54000/54000 [==============================] - 12s 214us/step - loss: 0.5882 - acc: 0.8070 - val_loss: 0.2462 - val_acc: 0.9172
Epoch 33/200
54000/54000 [==============================] - 10s 180us/step - loss: 0.5928 - acc: 0.8065 - val_loss: 0.2514 - val_acc: 0.9210
Epoch 34/200
54000/54000 [==============================] - 11s 201us/step - loss: 0.5876 - acc: 0.8075 - val_loss: 0.2559 - val_acc: 0.9147
Epoch 35/200
54000/54000 [==============================] - 10s 183us/step - loss: 0.6027 - acc: 0.8066 - val_loss: 0.2870 - val_acc: 0.9193
Epoch 36/200
54000/54000 [==============================] - 10s 177us/step - loss: 0.6020 - acc: 0.8040 - val_loss: 0.2552 - val_acc: 0.9175
Epoch 37/200
54000/54000 [==============================] - 17s 308us/step - loss: 0.5990 - acc: 0.8053 - val_loss: 0.2822 - val_acc: 0.9222
Epoch 38/200
54000/54000 [==============================] - 20s 365us/step - loss: 0.5844 - acc: 0.8090 - val_loss: 0.2283 - val_acc: 0.9257
Epoch 39/200
54000/54000 [==============================] - 15s 281us/step - loss: 0.5796 - acc: 0.8121 - val_loss: 0.2354 - val_acc: 0.9303
Epoch 40/200
54000/54000 [==============================] - 19s 343us/step - loss: 0.5653 - acc: 0.8144 - val_loss: 0.2490 - val_acc: 0.9193
Epoch 41/200
54000/54000 [==============================] - 16s 296us/step - loss: 0.5870 - acc: 0.8086 - val_loss: 0.2395 - val_acc: 0.9323
Epoch 42/200
54000/54000 [==============================] - 11s 202us/step - loss: 0.5687 - acc: 0.8153 - val_loss: 0.2338 - val_acc: 0.9222
Epoch 43/200
54000/54000 [==============================] - 11s 205us/step - loss: 0.5852 - acc: 0.8111 - val_loss: 0.2612 - val_acc: 0.9213
Epoch 44/200
54000/54000 [==============================] - 12s 221us/step - loss: 0.5817 - acc: 0.8131 - val_loss: 0.2301 - val_acc: 0.9272
Epoch 45/200
54000/54000 [==============================] - 11s 196us/step - loss: 0.5734 - acc: 0.8150 - val_loss: 0.2748 - val_acc: 0.9237
Epoch 46/200
54000/54000 [==============================] - 13s 248us/step - loss: 0.5827 - acc: 0.8111 - val_loss: 0.2043 - val_acc: 0.9353
Epoch 47/200
54000/54000 [==============================] - 17s 316us/step - loss: 0.5581 - acc: 0.8196 - val_loss: 0.3398 - val_acc: 0.9033
Epoch 48/200
54000/54000 [==============================] - 13s 236us/step - loss: 0.6056 - acc: 0.8069 - val_loss: 0.3289 - val_acc: 0.8918
Epoch 49/200
54000/54000 [==============================] - 12s 216us/step - loss: 0.6175 - acc: 0.8013 - val_loss: 0.3024 - val_acc: 0.9132
Epoch 50/200
54000/54000 [==============================] - 10s 193us/step - loss: 0.5912 - acc: 0.8078 - val_loss: 0.2511 - val_acc: 0.9192
Epoch 51/200
54000/54000 [==============================] - 11s 198us/step - loss: 0.5869 - acc: 0.8115 - val_loss: 0.2467 - val_acc: 0.9245
Epoch 52/200
54000/54000 [==============================] - 15s 269us/step - loss: 0.5581 - acc: 0.8181 - val_loss: 0.1997 - val_acc: 0.9370
Epoch 53/200
54000/54000 [==============================] - 15s 279us/step - loss: 0.5568 - acc: 0.8186 - val_loss: 0.2559 - val_acc: 0.9217
Epoch 54/200
54000/54000 [==============================] - 12s 231us/step - loss: 0.5641 - acc: 0.8164 - val_loss: 0.2035 - val_acc: 0.9397
Epoch 55/200
54000/54000 [==============================] - 17s 314us/step - loss: 0.5452 - acc: 0.8228 - val_loss: 0.2470 - val_acc: 0.9253
Epoch 56/200
54000/54000 [==============================] - 20s 364us/step - loss: 0.5678 - acc: 0.8163 - val_loss: 0.2091 - val_acc: 0.9335
Epoch 57/200
54000/54000 [==============================] - 18s 333us/step - loss: 0.5557 - acc: 0.8199 - val_loss: 0.4148 - val_acc: 0.8803
Epoch 58/200
54000/54000 [==============================] - 12s 229us/step - loss: 0.6423 - acc: 0.7935 - val_loss: 0.2579 - val_acc: 0.9165
Epoch 59/200
54000/54000 [==============================] - 11s 195us/step - loss: 0.5876 - acc: 0.8128 - val_loss: 0.2731 - val_acc: 0.9202
Epoch 60/200
54000/54000 [==============================] - 9s 173us/step - loss: 0.5699 - acc: 0.8158 - val_loss: 0.2963 - val_acc: 0.9083
Epoch 61/200
54000/54000 [==============================] - 8s 144us/step - loss: 0.5908 - acc: 0.8083 - val_loss: 0.2881 - val_acc: 0.9178
Epoch 62/200
54000/54000 [==============================] - 9s 161us/step - loss: 0.5750 - acc: 0.8125 - val_loss: 0.2800 - val_acc: 0.9125
Epoch 63/200
54000/54000 [==============================] - 9s 172us/step - loss: 0.5946 - acc: 0.8104 - val_loss: 0.2163 - val_acc: 0.9345
Epoch 64/200
54000/54000 [==============================] - 9s 175us/step - loss: 0.5468 - acc: 0.8225 - val_loss: 0.2073 - val_acc: 0.9368
Epoch 65/200
54000/54000 [==============================] - 11s 197us/step - loss: 0.5499 - acc: 0.8230 - val_loss: 0.2299 - val_acc: 0.9282
Epoch 66/200
54000/54000 [==============================] - 8s 153us/step - loss: 0.5468 - acc: 0.8209 - val_loss: 0.2055 - val_acc: 0.9367
Epoch 67/200
54000/54000 [==============================] - 10s 181us/step - loss: 0.5394 - acc: 0.8250 - val_loss: 0.2191 - val_acc: 0.9373
Epoch 68/200
54000/54000 [==============================] - 12s 226us/step - loss: 0.5422 - acc: 0.8246 - val_loss: 0.2908 - val_acc: 0.9088
Epoch 69/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5984 - acc: 0.8128 - val_loss: 0.2298 - val_acc: 0.9347
Epoch 70/200
54000/54000 [==============================] - 10s 183us/step - loss: 0.5471 - acc: 0.8219 - val_loss: 0.2653 - val_acc: 0.9192
Epoch 71/200
54000/54000 [==============================] - 12s 225us/step - loss: 0.5770 - acc: 0.8153 - val_loss: 0.3095 - val_acc: 0.9020
Epoch 72/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.5794 - acc: 0.8122 - val_loss: 0.2066 - val_acc: 0.9380
Epoch 73/200
54000/54000 [==============================] - 9s 169us/step - loss: 0.5396 - acc: 0.8241 - val_loss: 0.2613 - val_acc: 0.9237
Epoch 74/200
54000/54000 [==============================] - 8s 144us/step - loss: 0.5636 - acc: 0.8201 - val_loss: 0.3734 - val_acc: 0.8915
Epoch 75/200
54000/54000 [==============================] - 8s 154us/step - loss: 0.6015 - acc: 0.8052 - val_loss: 0.2395 - val_acc: 0.9308
Epoch 76/200
54000/54000 [==============================] - 10s 179us/step - loss: 0.5503 - acc: 0.8204 - val_loss: 0.3659 - val_acc: 0.8917
Epoch 77/200
54000/54000 [==============================] - 9s 164us/step - loss: 0.5817 - acc: 0.8118 - val_loss: 0.5141 - val_acc: 0.8452
Epoch 78/200
54000/54000 [==============================] - 8s 149us/step - loss: 0.6905 - acc: 0.7841 - val_loss: 0.2811 - val_acc: 0.9132
Epoch 79/200
54000/54000 [==============================] - 9s 165us/step - loss: 0.5666 - acc: 0.8168 - val_loss: 0.4576 - val_acc: 0.8583
Epoch 80/200
54000/54000 [==============================] - 8s 153us/step - loss: 0.6263 - acc: 0.7993 - val_loss: 0.1974 - val_acc: 0.9388
Epoch 81/200
54000/54000 [==============================] - 9s 175us/step - loss: 0.5357 - acc: 0.8259 - val_loss: 0.2161 - val_acc: 0.9337
Epoch 82/200
54000/54000 [==============================] - 9s 175us/step - loss: 0.5448 - acc: 0.8244 - val_loss: 0.3534 - val_acc: 0.8918
Epoch 83/200
54000/54000 [==============================] - 9s 163us/step - loss: 0.5798 - acc: 0.8107 - val_loss: 0.3855 - val_acc: 0.8807
Epoch 84/200
54000/54000 [==============================] - 8s 153us/step - loss: 0.6349 - acc: 0.8003 - val_loss: 0.2128 - val_acc: 0.9350
Epoch 85/200
54000/54000 [==============================] - 7s 128us/step - loss: 0.5422 - acc: 0.8246 - val_loss: 0.3271 - val_acc: 0.8968
Epoch 86/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5838 - acc: 0.8141 - val_loss: 0.2502 - val_acc: 0.9208
Epoch 87/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5604 - acc: 0.8180 - val_loss: 0.2908 - val_acc: 0.9148
Epoch 88/200
54000/54000 [==============================] - 8s 142us/step - loss: 0.5759 - acc: 0.8129 - val_loss: 0.3673 - val_acc: 0.8818
Epoch 89/200
54000/54000 [==============================] - 9s 159us/step - loss: 0.6072 - acc: 0.8052 - val_loss: 0.2121 - val_acc: 0.9363
Epoch 90/200
54000/54000 [==============================] - 9s 162us/step - loss: 0.5465 - acc: 0.8238 - val_loss: 0.2058 - val_acc: 0.9368
Epoch 91/200
54000/54000 [==============================] - 10s 186us/step - loss: 0.5377 - acc: 0.8276 - val_loss: 0.5367 - val_acc: 0.8370
Epoch 92/200
54000/54000 [==============================] - 10s 177us/step - loss: 0.6635 - acc: 0.7921 - val_loss: 0.3039 - val_acc: 0.9072
Epoch 93/200
54000/54000 [==============================] - 9s 171us/step - loss: 0.5712 - acc: 0.8137 - val_loss: 0.2773 - val_acc: 0.9183
Epoch 94/200
54000/54000 [==============================] - 8s 140us/step - loss: 0.5605 - acc: 0.8197 - val_loss: 0.4018 - val_acc: 0.8797
Epoch 95/200
54000/54000 [==============================] - 9s 164us/step - loss: 0.6062 - acc: 0.8059 - val_loss: 0.2626 - val_acc: 0.9202
Epoch 96/200
54000/54000 [==============================] - 9s 172us/step - loss: 0.5763 - acc: 0.8172 - val_loss: 0.1947 - val_acc: 0.9417
Epoch 97/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.5350 - acc: 0.8265 - val_loss: 0.2227 - val_acc: 0.9350
Epoch 98/200
54000/54000 [==============================] - 8s 156us/step - loss: 0.5465 - acc: 0.8258 - val_loss: 0.2052 - val_acc: 0.9357
Epoch 99/200
54000/54000 [==============================] - 8s 154us/step - loss: 0.5324 - acc: 0.8280 - val_loss: 0.2124 - val_acc: 0.9393
Epoch 100/200
54000/54000 [==============================] - 8s 155us/step - loss: 0.5364 - acc: 0.8291 - val_loss: 0.1994 - val_acc: 0.9378
Epoch 101/200
54000/54000 [==============================] - 9s 162us/step - loss: 0.5339 - acc: 0.8271 - val_loss: 0.2096 - val_acc: 0.9382
Epoch 102/200
54000/54000 [==============================] - 9s 174us/step - loss: 0.5410 - acc: 0.8283 - val_loss: 0.2305 - val_acc: 0.9338
Epoch 103/200
54000/54000 [==============================] - 8s 142us/step - loss: 0.5383 - acc: 0.8256 - val_loss: 0.3488 - val_acc: 0.9008
Epoch 104/200
54000/54000 [==============================] - 8s 140us/step - loss: 0.5918 - acc: 0.8106 - val_loss: 0.3120 - val_acc: 0.9008
Epoch 105/200
54000/54000 [==============================] - 9s 163us/step - loss: 0.5821 - acc: 0.8159 - val_loss: 0.2411 - val_acc: 0.9292
Epoch 106/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.5486 - acc: 0.8240 - val_loss: 0.2756 - val_acc: 0.9120
Epoch 107/200
54000/54000 [==============================] - 8s 157us/step - loss: 0.5584 - acc: 0.8200 - val_loss: 0.2550 - val_acc: 0.9177
Epoch 108/200
54000/54000 [==============================] - 11s 207us/step - loss: 0.5602 - acc: 0.8202 - val_loss: 0.2627 - val_acc: 0.9170
Epoch 109/200
54000/54000 [==============================] - 15s 276us/step - loss: 0.5523 - acc: 0.8235 - val_loss: 0.2847 - val_acc: 0.9182
Epoch 110/200
54000/54000 [==============================] - 10s 186us/step - loss: 0.5672 - acc: 0.8174 - val_loss: 0.2042 - val_acc: 0.9362
Epoch 111/200
54000/54000 [==============================] - 14s 251us/step - loss: 0.5392 - acc: 0.8264 - val_loss: 0.2582 - val_acc: 0.9210
Epoch 112/200
54000/54000 [==============================] - 9s 172us/step - loss: 0.5570 - acc: 0.8216 - val_loss: 0.2599 - val_acc: 0.9210
Epoch 113/200
54000/54000 [==============================] - 11s 196us/step - loss: 0.5532 - acc: 0.8207 - val_loss: 0.2561 - val_acc: 0.9153
Epoch 114/200
54000/54000 [==============================] - 10s 182us/step - loss: 0.5579 - acc: 0.8212 - val_loss: 0.2083 - val_acc: 0.9332
Epoch 115/200
54000/54000 [==============================] - 9s 171us/step - loss: 0.5380 - acc: 0.8269 - val_loss: 0.3357 - val_acc: 0.9043
Epoch 116/200
54000/54000 [==============================] - 12s 213us/step - loss: 0.6166 - acc: 0.8102 - val_loss: 0.3330 - val_acc: 0.9023
Epoch 117/200
54000/54000 [==============================] - 10s 191us/step - loss: 0.5693 - acc: 0.8138 - val_loss: 0.2202 - val_acc: 0.9345
Epoch 118/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.5379 - acc: 0.8271 - val_loss: 0.4368 - val_acc: 0.8708
Epoch 119/200
54000/54000 [==============================] - 7s 130us/step - loss: 0.6577 - acc: 0.7851 - val_loss: 0.3179 - val_acc: 0.9037
Epoch 120/200
54000/54000 [==============================] - 6s 118us/step - loss: 0.5965 - acc: 0.8123 - val_loss: 0.1930 - val_acc: 0.9412
Epoch 121/200
54000/54000 [==============================] - 8s 146us/step - loss: 0.5244 - acc: 0.8322 - val_loss: 0.4474 - val_acc: 0.8725
Epoch 122/200
54000/54000 [==============================] - 10s 179us/step - loss: 0.6087 - acc: 0.8103 - val_loss: 0.4378 - val_acc: 0.8718
Epoch 123/200
54000/54000 [==============================] - 8s 141us/step - loss: 0.6253 - acc: 0.7974 - val_loss: 0.3291 - val_acc: 0.9037
Epoch 124/200
54000/54000 [==============================] - 9s 171us/step - loss: 0.6002 - acc: 0.8144 - val_loss: 0.4941 - val_acc: 0.8537
Epoch 125/200
54000/54000 [==============================] - 7s 135us/step - loss: 0.6677 - acc: 0.7816 - val_loss: 0.3775 - val_acc: 0.8933
Epoch 126/200
54000/54000 [==============================] - 7s 127us/step - loss: 0.6198 - acc: 0.8007 - val_loss: 0.3835 - val_acc: 0.8918
Epoch 127/200
54000/54000 [==============================] - 8s 148us/step - loss: 0.6007 - acc: 0.8054 - val_loss: 0.9414 - val_acc: 0.7368
Epoch 128/200
54000/54000 [==============================] - 9s 161us/step - loss: 0.8110 - acc: 0.7734 - val_loss: 0.3031 - val_acc: 0.9087
Epoch 129/200
54000/54000 [==============================] - 6s 116us/step - loss: 0.5593 - acc: 0.8190 - val_loss: 0.3126 - val_acc: 0.9073
Epoch 130/200
54000/54000 [==============================] - 8s 151us/step - loss: 0.5919 - acc: 0.8179 - val_loss: 0.3175 - val_acc: 0.9030
Epoch 131/200
54000/54000 [==============================] - 9s 164us/step - loss: 0.5761 - acc: 0.8146 - val_loss: 0.4523 - val_acc: 0.8782
Epoch 132/200
54000/54000 [==============================] - 9s 174us/step - loss: 0.6465 - acc: 0.8079 - val_loss: 0.3022 - val_acc: 0.9115
Epoch 133/200
54000/54000 [==============================] - 8s 147us/step - loss: 0.5766 - acc: 0.8159 - val_loss: 0.2019 - val_acc: 0.9418
Epoch 134/200
54000/54000 [==============================] - 11s 206us/step - loss: 0.5376 - acc: 0.8293 - val_loss: 0.2199 - val_acc: 0.9338
Epoch 135/200
54000/54000 [==============================] - 11s 196us/step - loss: 0.5401 - acc: 0.8250 - val_loss: 0.2405 - val_acc: 0.9335
Epoch 136/200
54000/54000 [==============================] - 9s 168us/step - loss: 0.5746 - acc: 0.8220 - val_loss: 0.3336 - val_acc: 0.9070
Epoch 137/200
54000/54000 [==============================] - 10s 192us/step - loss: 0.5943 - acc: 0.8072 - val_loss: 0.3385 - val_acc: 0.9052
Epoch 138/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5817 - acc: 0.8125 - val_loss: 0.2107 - val_acc: 0.9393
Epoch 139/200
54000/54000 [==============================] - 8s 149us/step - loss: 0.5327 - acc: 0.8294 - val_loss: 0.3317 - val_acc: 0.9025
Epoch 140/200
54000/54000 [==============================] - 10s 180us/step - loss: 0.6149 - acc: 0.8104 - val_loss: 0.6000 - val_acc: 0.8257
Epoch 141/200
54000/54000 [==============================] - 7s 136us/step - loss: 0.6803 - acc: 0.7778 - val_loss: 0.3049 - val_acc: 0.9103
Epoch 142/200
54000/54000 [==============================] - 9s 166us/step - loss: 0.5934 - acc: 0.8168 - val_loss: 0.3723 - val_acc: 0.8927
Epoch 143/200
54000/54000 [==============================] - 9s 168us/step - loss: 0.5882 - acc: 0.8113 - val_loss: 0.3675 - val_acc: 0.8935
Epoch 144/200
54000/54000 [==============================] - 9s 163us/step - loss: 0.6294 - acc: 0.8083 - val_loss: 0.3813 - val_acc: 0.8920
Epoch 145/200
54000/54000 [==============================] - 9s 171us/step - loss: 0.6195 - acc: 0.8000 - val_loss: 0.6312 - val_acc: 0.8320
Epoch 146/200
54000/54000 [==============================] - 7s 127us/step - loss: 0.6764 - acc: 0.8012 - val_loss: 0.5133 - val_acc: 0.8597
Epoch 147/200
54000/54000 [==============================] - 10s 182us/step - loss: 0.6427 - acc: 0.7910 - val_loss: 0.4616 - val_acc: 0.8737
Epoch 148/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.6698 - acc: 0.8017 - val_loss: 0.7324 - val_acc: 0.7892
Epoch 149/200
54000/54000 [==============================] - 8s 152us/step - loss: 0.6809 - acc: 0.7779 - val_loss: 0.2204 - val_acc: 0.9318
Epoch 150/200
54000/54000 [==============================] - 14s 254us/step - loss: 0.5464 - acc: 0.8264 - val_loss: 0.4623 - val_acc: 0.8755
Epoch 151/200
54000/54000 [==============================] - 10s 188us/step - loss: 0.6444 - acc: 0.7929 - val_loss: 0.2392 - val_acc: 0.9280
Epoch 152/200
54000/54000 [==============================] - 16s 299us/step - loss: 0.5498 - acc: 0.8267 - val_loss: 0.3395 - val_acc: 0.9107
Epoch 153/200
54000/54000 [==============================] - 10s 192us/step - loss: 0.5689 - acc: 0.8187 - val_loss: 0.2745 - val_acc: 0.9220
Epoch 154/200
54000/54000 [==============================] - 13s 238us/step - loss: 0.5471 - acc: 0.8264 - val_loss: 0.6109 - val_acc: 0.8250
Epoch 155/200
54000/54000 [==============================] - 12s 221us/step - loss: 0.6744 - acc: 0.7819 - val_loss: 0.5190 - val_acc: 0.8648
Epoch 156/200
54000/54000 [==============================] - 8s 151us/step - loss: 0.6308 - acc: 0.8147 - val_loss: 0.7427 - val_acc: 0.7740
Epoch 157/200
54000/54000 [==============================] - 13s 240us/step - loss: 0.6927 - acc: 0.7759 - val_loss: 0.8837 - val_acc: 0.7750
Epoch 158/200
54000/54000 [==============================] - 14s 253us/step - loss: 0.7516 - acc: 0.7883 - val_loss: 0.2063 - val_acc: 0.9373
Epoch 159/200
54000/54000 [==============================] - 13s 237us/step - loss: 0.5322 - acc: 0.8304 - val_loss: 0.3463 - val_acc: 0.9025
Epoch 160/200
54000/54000 [==============================] - 13s 241us/step - loss: 0.6216 - acc: 0.8104 - val_loss: 0.4496 - val_acc: 0.8767
Epoch 161/200
54000/54000 [==============================] - 14s 258us/step - loss: 0.6171 - acc: 0.8027 - val_loss: 0.4166 - val_acc: 0.8830
Epoch 162/200
54000/54000 [==============================] - 13s 244us/step - loss: 0.5866 - acc: 0.8117 - val_loss: 0.4412 - val_acc: 0.8808
Epoch 163/200
54000/54000 [==============================] - 14s 253us/step - loss: 0.6334 - acc: 0.7979 - val_loss: 0.4461 - val_acc: 0.8872
Epoch 164/200
54000/54000 [==============================] - 14s 252us/step - loss: 0.6357 - acc: 0.7965 - val_loss: 0.4822 - val_acc: 0.8665
Epoch 165/200
54000/54000 [==============================] - 15s 286us/step - loss: 0.5979 - acc: 0.8064 - val_loss: 0.2993 - val_acc: 0.9133
Epoch 166/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.5924 - acc: 0.8205 - val_loss: 0.5238 - val_acc: 0.8533
Epoch 167/200
54000/54000 [==============================] - 10s 176us/step - loss: 0.6319 - acc: 0.7977 - val_loss: 0.8002 - val_acc: 0.8072
Epoch 168/200
54000/54000 [==============================] - 9s 170us/step - loss: 0.7293 - acc: 0.7947 - val_loss: 0.3961 - val_acc: 0.8947
Epoch 169/200
54000/54000 [==============================] - 9s 167us/step - loss: 0.6201 - acc: 0.8018 - val_loss: 0.5964 - val_acc: 0.8485
Epoch 170/200
54000/54000 [==============================] - 9s 168us/step - loss: 0.6604 - acc: 0.8061 - val_loss: 0.4253 - val_acc: 0.8822
Epoch 171/200
54000/54000 [==============================] - 10s 192us/step - loss: 0.6276 - acc: 0.7972 - val_loss: 0.2758 - val_acc: 0.9238
Epoch 172/200
54000/54000 [==============================] - 13s 250us/step - loss: 0.5802 - acc: 0.8229 - val_loss: 0.5523 - val_acc: 0.8425
Epoch 173/200
54000/54000 [==============================] - 8s 157us/step - loss: 0.6240 - acc: 0.7963 - val_loss: 0.5082 - val_acc: 0.8690
Epoch 174/200
54000/54000 [==============================] - 10s 187us/step - loss: 0.7204 - acc: 0.7995 - val_loss: 0.2203 - val_acc: 0.9372
Epoch 175/200
54000/54000 [==============================] - 9s 165us/step - loss: 0.5457 - acc: 0.8244 - val_loss: 0.6313 - val_acc: 0.8167
Epoch 176/200
54000/54000 [==============================] - 9s 164us/step - loss: 0.6973 - acc: 0.7731 - val_loss: 0.6232 - val_acc: 0.8205
Epoch 177/200
54000/54000 [==============================] - 9s 173us/step - loss: 0.6489 - acc: 0.7891 - val_loss: 0.2650 - val_acc: 0.9230
Epoch 178/200
54000/54000 [==============================] - 8s 151us/step - loss: 0.5760 - acc: 0.8217 - val_loss: 0.7826 - val_acc: 0.7667
Epoch 179/200
54000/54000 [==============================] - 7s 128us/step - loss: 0.7262 - acc: 0.7635 - val_loss: 0.2920 - val_acc: 0.9187
Epoch 180/200
54000/54000 [==============================] - 9s 161us/step - loss: 0.6067 - acc: 0.8170 - val_loss: 0.2349 - val_acc: 0.9310
Epoch 181/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5637 - acc: 0.8242 - val_loss: 0.4128 - val_acc: 0.8975
Epoch 182/200
54000/54000 [==============================] - 6s 120us/step - loss: 0.6823 - acc: 0.8045 - val_loss: 0.2637 - val_acc: 0.9278
Epoch 183/200
54000/54000 [==============================] - 9s 167us/step - loss: 0.5810 - acc: 0.8249 - val_loss: 0.2731 - val_acc: 0.9212
Epoch 184/200
54000/54000 [==============================] - 8s 150us/step - loss: 0.5698 - acc: 0.8246 - val_loss: 0.3098 - val_acc: 0.9108
Epoch 185/200
54000/54000 [==============================] - 7s 125us/step - loss: 0.6092 - acc: 0.8141 - val_loss: 0.2550 - val_acc: 0.9267
Epoch 186/200
54000/54000 [==============================] - 9s 170us/step - loss: 0.5734 - acc: 0.8214 - val_loss: 0.2929 - val_acc: 0.9152
Epoch 187/200
54000/54000 [==============================] - 9s 160us/step - loss: 0.6176 - acc: 0.8131 - val_loss: 0.3180 - val_acc: 0.9118
Epoch 188/200
54000/54000 [==============================] - 7s 132us/step - loss: 0.5682 - acc: 0.8167 - val_loss: 0.5216 - val_acc: 0.8618
Epoch 189/200
54000/54000 [==============================] - 8s 143us/step - loss: 0.6770 - acc: 0.8007 - val_loss: 0.4549 - val_acc: 0.8767
Epoch 190/200
54000/54000 [==============================] - 8s 140us/step - loss: 0.6046 - acc: 0.8046 - val_loss: 0.2460 - val_acc: 0.9313
Epoch 191/200
54000/54000 [==============================] - 9s 160us/step - loss: 0.5398 - acc: 0.8283 - val_loss: 0.2320 - val_acc: 0.9327
Epoch 192/200
54000/54000 [==============================] - 8s 144us/step - loss: 0.5511 - acc: 0.8262 - val_loss: 0.2271 - val_acc: 0.9335
Epoch 193/200
54000/54000 [==============================] - 8s 148us/step - loss: 0.5522 - acc: 0.8257 - val_loss: 0.4769 - val_acc: 0.8762
Epoch 194/200
54000/54000 [==============================] - 8s 142us/step - loss: 0.6079 - acc: 0.8059 - val_loss: 0.2280 - val_acc: 0.9330
Epoch 195/200
54000/54000 [==============================] - 8s 141us/step - loss: 0.5493 - acc: 0.8281 - val_loss: 0.3418 - val_acc: 0.9043
Epoch 196/200
54000/54000 [==============================] - 8s 144us/step - loss: 0.5860 - acc: 0.8190 - val_loss: 0.2448 - val_acc: 0.9273
Epoch 197/200
54000/54000 [==============================] - 7s 139us/step - loss: 0.5664 - acc: 0.8221 - val_loss: 0.3728 - val_acc: 0.8957
Epoch 198/200
54000/54000 [==============================] - 8s 145us/step - loss: 0.5794 - acc: 0.8130 - val_loss: 0.5258 - val_acc: 0.8577
Epoch 199/200
54000/54000 [==============================] - 7s 136us/step - loss: 0.6231 - acc: 0.8095 - val_loss: 0.6859 - val_acc: 0.8315
Epoch 200/200
54000/54000 [==============================] - 7s 135us/step - loss: 0.7339 - acc: 0.7913 - val_loss: 0.3871 - val_acc: 0.8950

学習が終わったところで、学習過程を可視化してみます。

In [119]:
plt.figure()
plt.plot(mm3_lp.epoch, np.array(mm3_lp.history['acc']),label='Train $10^{-3}$', c="g", ls="-.")
plt.plot(mm3_lp.epoch, np.array(mm3_lp.history['val_acc']),label='Valid. $10^{-3}$', c="g", ls="-")
plt.xlabel('Epoch')
plt.ylabel('Acc')
plt.grid()
plt.legend()
Out[119]:
<matplotlib.legend.Legend at 0x15f4a8978>

Validation data の正答率(val_acc)が重要になります(学習に使用していないデータの正答率のため)。そして、val_accは、75%〜95%くらいの範囲にあることがわかります。また、epochが多い方がval_accが必ずしも高くないことが見て取れます。したがって、epoch=200のモデルが最適ではないことになります。そのため、val_accが最大となるepochを求めてみます。

In [126]:
val_acc = np.array(mm3_lp.history['val_acc'])
ep_argmax = np.argmax(val_acc)
best_epoch = ep_argmax + 1
print("最適なepoch: ", best_epoch)
print("そのときの正答率(val_acc)", val_acc[ep_argmax])
# 配列は0スタートだが、epochは1スタートのため、1を足す

print("\n最後のepoch: ", len(val_acc))
print("最後の正答率(val_acc)", val_acc[-1])
最適なepoch:  133
そのときの正答率(val_acc) 0.9418333238363266

最後のepoch:  200
最後の正答率(val_acc) 0.8949999980926514

一番良いepochを採用した場合と、最後まで学習させた場合で、val_accの間に5%近くの差があることがわかりました。したがって、epochは133を採用すべきと判断できます。このepochを用いて、モデルを作ってみましょう。

In [127]:
epochs = best_epoch # 学習回数
se = 0 # 乱数シード番号(再現したいモデルと同じ値)

print("\n ### モデル3: 学習係数=10^-3")
os.environ['PYTHONHASHSEED'] = str(se)
np.random.seed(se)
mm3_best = Sequential()
mm3_best = mod_lr(mm3_best, 10**-3)
mm3_lp_best = mm3_best.fit(x_train_cnn, y_train, batch_size=batch_size, 
               epochs=epochs, verbose=1, validation_split=0.1)
 ### モデル3: 学習係数=10^-3
Train on 54000 samples, validate on 6000 samples
Epoch 1/133
54000/54000 [==============================] - 16s 304us/step - loss: 1.4584 - acc: 0.4981 - val_loss: 0.6115 - val_acc: 0.8527
Epoch 2/133
54000/54000 [==============================] - 8s 150us/step - loss: 0.8982 - acc: 0.7039 - val_loss: 0.3623 - val_acc: 0.8850
Epoch 3/133
54000/54000 [==============================] - 7s 135us/step - loss: 0.8019 - acc: 0.7340 - val_loss: 0.3862 - val_acc: 0.8917
Epoch 4/133
54000/54000 [==============================] - 8s 142us/step - loss: 0.7743 - acc: 0.7433 - val_loss: 0.3178 - val_acc: 0.9010
Epoch 5/133
54000/54000 [==============================] - 12s 226us/step - loss: 0.7446 - acc: 0.7521 - val_loss: 0.3545 - val_acc: 0.9002
Epoch 6/133
54000/54000 [==============================] - 10s 187us/step - loss: 0.7317 - acc: 0.7582 - val_loss: 0.3491 - val_acc: 0.9010
Epoch 7/133
54000/54000 [==============================] - 8s 151us/step - loss: 0.7205 - acc: 0.7627 - val_loss: 0.3206 - val_acc: 0.9047
Epoch 8/133
54000/54000 [==============================] - 10s 185us/step - loss: 0.7021 - acc: 0.7682 - val_loss: 0.3467 - val_acc: 0.9070
Epoch 9/133
54000/54000 [==============================] - 10s 176us/step - loss: 0.6919 - acc: 0.7725 - val_loss: 0.2770 - val_acc: 0.9070
Epoch 10/133
54000/54000 [==============================] - 9s 161us/step - loss: 0.6836 - acc: 0.7745 - val_loss: 0.3125 - val_acc: 0.9048
Epoch 11/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.6761 - acc: 0.7756 - val_loss: 0.3086 - val_acc: 0.9123
Epoch 12/133
54000/54000 [==============================] - 7s 137us/step - loss: 0.6624 - acc: 0.7815 - val_loss: 0.2953 - val_acc: 0.9215
Epoch 13/133
54000/54000 [==============================] - 8s 153us/step - loss: 0.6492 - acc: 0.7868 - val_loss: 0.3041 - val_acc: 0.9108
Epoch 14/133
54000/54000 [==============================] - 9s 170us/step - loss: 0.6510 - acc: 0.7866 - val_loss: 0.2675 - val_acc: 0.9135
Epoch 15/133
54000/54000 [==============================] - 8s 156us/step - loss: 0.6474 - acc: 0.7859 - val_loss: 0.2924 - val_acc: 0.9192
Epoch 16/133
54000/54000 [==============================] - 9s 169us/step - loss: 0.6299 - acc: 0.7930 - val_loss: 0.2415 - val_acc: 0.9205
Epoch 17/133
54000/54000 [==============================] - 9s 166us/step - loss: 0.6287 - acc: 0.7954 - val_loss: 0.2836 - val_acc: 0.9192
Epoch 18/133
54000/54000 [==============================] - 9s 172us/step - loss: 0.6244 - acc: 0.7965 - val_loss: 0.2936 - val_acc: 0.9007
Epoch 19/133
54000/54000 [==============================] - 8s 156us/step - loss: 0.6592 - acc: 0.7862 - val_loss: 0.2687 - val_acc: 0.9252
Epoch 20/133
54000/54000 [==============================] - 7s 125us/step - loss: 0.6112 - acc: 0.7995 - val_loss: 0.2419 - val_acc: 0.9243
Epoch 21/133
54000/54000 [==============================] - 7s 137us/step - loss: 0.6086 - acc: 0.7994 - val_loss: 0.2767 - val_acc: 0.9248
Epoch 22/133
54000/54000 [==============================] - 6s 117us/step - loss: 0.6050 - acc: 0.8017 - val_loss: 0.2503 - val_acc: 0.9153
Epoch 23/133
54000/54000 [==============================] - 7s 131us/step - loss: 0.6099 - acc: 0.8008 - val_loss: 0.3330 - val_acc: 0.8925
Epoch 24/133
54000/54000 [==============================] - 8s 149us/step - loss: 0.6489 - acc: 0.7848 - val_loss: 0.2343 - val_acc: 0.9227
Epoch 25/133
54000/54000 [==============================] - 9s 160us/step - loss: 0.6063 - acc: 0.8022 - val_loss: 0.2814 - val_acc: 0.9245
Epoch 26/133
54000/54000 [==============================] - 8s 150us/step - loss: 0.6071 - acc: 0.8019 - val_loss: 0.2263 - val_acc: 0.9270
Epoch 27/133
54000/54000 [==============================] - 7s 123us/step - loss: 0.5928 - acc: 0.8056 - val_loss: 0.2634 - val_acc: 0.9283
Epoch 28/133
54000/54000 [==============================] - 7s 122us/step - loss: 0.5913 - acc: 0.8081 - val_loss: 0.2519 - val_acc: 0.9178
Epoch 29/133
54000/54000 [==============================] - 8s 149us/step - loss: 0.6016 - acc: 0.8042 - val_loss: 0.2881 - val_acc: 0.9158
Epoch 30/133
54000/54000 [==============================] - 8s 142us/step - loss: 0.5981 - acc: 0.8052 - val_loss: 0.2598 - val_acc: 0.9147
Epoch 31/133
54000/54000 [==============================] - 8s 143us/step - loss: 0.6049 - acc: 0.8041 - val_loss: 0.2665 - val_acc: 0.9238
Epoch 32/133
54000/54000 [==============================] - 7s 130us/step - loss: 0.5882 - acc: 0.8070 - val_loss: 0.2462 - val_acc: 0.9172
Epoch 33/133
54000/54000 [==============================] - 7s 132us/step - loss: 0.5928 - acc: 0.8065 - val_loss: 0.2514 - val_acc: 0.9210
Epoch 34/133
54000/54000 [==============================] - 8s 139us/step - loss: 0.5876 - acc: 0.8075 - val_loss: 0.2559 - val_acc: 0.9147
Epoch 35/133
54000/54000 [==============================] - 7s 129us/step - loss: 0.6027 - acc: 0.8066 - val_loss: 0.2870 - val_acc: 0.9193
Epoch 36/133
54000/54000 [==============================] - 7s 133us/step - loss: 0.6020 - acc: 0.8040 - val_loss: 0.2552 - val_acc: 0.9175
Epoch 37/133
54000/54000 [==============================] - 6s 112us/step - loss: 0.5990 - acc: 0.8053 - val_loss: 0.2822 - val_acc: 0.9222
Epoch 38/133
54000/54000 [==============================] - 6s 113us/step - loss: 0.5844 - acc: 0.8090 - val_loss: 0.2283 - val_acc: 0.9257
Epoch 39/133
54000/54000 [==============================] - 6s 113us/step - loss: 0.5796 - acc: 0.8121 - val_loss: 0.2354 - val_acc: 0.9303
Epoch 40/133
54000/54000 [==============================] - 7s 127us/step - loss: 0.5653 - acc: 0.8144 - val_loss: 0.2490 - val_acc: 0.9193
Epoch 41/133
54000/54000 [==============================] - 7s 122us/step - loss: 0.5870 - acc: 0.8086 - val_loss: 0.2395 - val_acc: 0.9323
Epoch 42/133
54000/54000 [==============================] - 7s 131us/step - loss: 0.5687 - acc: 0.8153 - val_loss: 0.2338 - val_acc: 0.9222
Epoch 43/133
54000/54000 [==============================] - 7s 137us/step - loss: 0.5852 - acc: 0.8111 - val_loss: 0.2612 - val_acc: 0.9213
Epoch 44/133
54000/54000 [==============================] - 8s 142us/step - loss: 0.5817 - acc: 0.8131 - val_loss: 0.2301 - val_acc: 0.9272
Epoch 45/133
54000/54000 [==============================] - 7s 136us/step - loss: 0.5734 - acc: 0.8150 - val_loss: 0.2748 - val_acc: 0.9237
Epoch 46/133
54000/54000 [==============================] - 6s 112us/step - loss: 0.5827 - acc: 0.8111 - val_loss: 0.2043 - val_acc: 0.9353
Epoch 47/133
54000/54000 [==============================] - 7s 135us/step - loss: 0.5581 - acc: 0.8196 - val_loss: 0.3398 - val_acc: 0.9033
Epoch 48/133
54000/54000 [==============================] - 14s 268us/step - loss: 0.6056 - acc: 0.8069 - val_loss: 0.3289 - val_acc: 0.8918
Epoch 49/133
54000/54000 [==============================] - 10s 194us/step - loss: 0.6175 - acc: 0.8013 - val_loss: 0.3024 - val_acc: 0.9132
Epoch 50/133
54000/54000 [==============================] - 8s 152us/step - loss: 0.5912 - acc: 0.8078 - val_loss: 0.2511 - val_acc: 0.9192
Epoch 51/133
54000/54000 [==============================] - 8s 148us/step - loss: 0.5869 - acc: 0.8115 - val_loss: 0.2467 - val_acc: 0.9245
Epoch 52/133
54000/54000 [==============================] - 9s 164us/step - loss: 0.5581 - acc: 0.8181 - val_loss: 0.1997 - val_acc: 0.9370
Epoch 53/133
54000/54000 [==============================] - 9s 162us/step - loss: 0.5568 - acc: 0.8186 - val_loss: 0.2559 - val_acc: 0.9217
Epoch 54/133
54000/54000 [==============================] - 8s 156us/step - loss: 0.5641 - acc: 0.8164 - val_loss: 0.2035 - val_acc: 0.9397
Epoch 55/133
54000/54000 [==============================] - 6s 115us/step - loss: 0.5452 - acc: 0.8228 - val_loss: 0.2470 - val_acc: 0.9253
Epoch 56/133
54000/54000 [==============================] - 6s 109us/step - loss: 0.5678 - acc: 0.8163 - val_loss: 0.2091 - val_acc: 0.9335
Epoch 57/133
54000/54000 [==============================] - 6s 110us/step - loss: 0.5557 - acc: 0.8199 - val_loss: 0.4148 - val_acc: 0.8803
Epoch 58/133
54000/54000 [==============================] - 7s 128us/step - loss: 0.6423 - acc: 0.7935 - val_loss: 0.2579 - val_acc: 0.9165
Epoch 59/133
54000/54000 [==============================] - 9s 163us/step - loss: 0.5876 - acc: 0.8128 - val_loss: 0.2731 - val_acc: 0.9202
Epoch 60/133
54000/54000 [==============================] - 7s 133us/step - loss: 0.5699 - acc: 0.8158 - val_loss: 0.2963 - val_acc: 0.9083
Epoch 61/133
54000/54000 [==============================] - 7s 138us/step - loss: 0.5908 - acc: 0.8083 - val_loss: 0.2881 - val_acc: 0.9178
Epoch 62/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.5750 - acc: 0.8125 - val_loss: 0.2800 - val_acc: 0.9125
Epoch 63/133
54000/54000 [==============================] - 7s 137us/step - loss: 0.5946 - acc: 0.8104 - val_loss: 0.2163 - val_acc: 0.9345
Epoch 64/133
54000/54000 [==============================] - 8s 154us/step - loss: 0.5468 - acc: 0.8225 - val_loss: 0.2073 - val_acc: 0.9368
Epoch 65/133
54000/54000 [==============================] - 7s 135us/step - loss: 0.5499 - acc: 0.8230 - val_loss: 0.2299 - val_acc: 0.9282
Epoch 66/133
54000/54000 [==============================] - 8s 146us/step - loss: 0.5468 - acc: 0.8209 - val_loss: 0.2055 - val_acc: 0.9367
Epoch 67/133
54000/54000 [==============================] - 7s 130us/step - loss: 0.5394 - acc: 0.8250 - val_loss: 0.2191 - val_acc: 0.9373
Epoch 68/133
54000/54000 [==============================] - 8s 147us/step - loss: 0.5422 - acc: 0.8246 - val_loss: 0.2908 - val_acc: 0.9088
Epoch 69/133
54000/54000 [==============================] - 7s 134us/step - loss: 0.5984 - acc: 0.8128 - val_loss: 0.2298 - val_acc: 0.9347
Epoch 70/133
54000/54000 [==============================] - 7s 124us/step - loss: 0.5471 - acc: 0.8219 - val_loss: 0.2653 - val_acc: 0.9192
Epoch 71/133
54000/54000 [==============================] - 7s 130us/step - loss: 0.5770 - acc: 0.8153 - val_loss: 0.3095 - val_acc: 0.9020
Epoch 72/133
54000/54000 [==============================] - 9s 173us/step - loss: 0.5794 - acc: 0.8122 - val_loss: 0.2066 - val_acc: 0.9380
Epoch 73/133
54000/54000 [==============================] - 7s 122us/step - loss: 0.5396 - acc: 0.8241 - val_loss: 0.2613 - val_acc: 0.9237
Epoch 74/133
54000/54000 [==============================] - 7s 132us/step - loss: 0.5636 - acc: 0.8201 - val_loss: 0.3734 - val_acc: 0.8915
Epoch 75/133
54000/54000 [==============================] - 8s 140us/step - loss: 0.6015 - acc: 0.8052 - val_loss: 0.2395 - val_acc: 0.9308
Epoch 76/133
54000/54000 [==============================] - 9s 161us/step - loss: 0.5503 - acc: 0.8204 - val_loss: 0.3659 - val_acc: 0.8917
Epoch 77/133
54000/54000 [==============================] - 7s 134us/step - loss: 0.5817 - acc: 0.8118 - val_loss: 0.5141 - val_acc: 0.8452
Epoch 78/133
54000/54000 [==============================] - 7s 130us/step - loss: 0.6905 - acc: 0.7841 - val_loss: 0.2811 - val_acc: 0.9132
Epoch 79/133
54000/54000 [==============================] - 9s 165us/step - loss: 0.5666 - acc: 0.8168 - val_loss: 0.4576 - val_acc: 0.8583
Epoch 80/133
54000/54000 [==============================] - 9s 161us/step - loss: 0.6263 - acc: 0.7993 - val_loss: 0.1974 - val_acc: 0.9388
Epoch 81/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.5357 - acc: 0.8259 - val_loss: 0.2161 - val_acc: 0.9337
Epoch 82/133
54000/54000 [==============================] - 8s 146us/step - loss: 0.5448 - acc: 0.8244 - val_loss: 0.3534 - val_acc: 0.8918
Epoch 83/133
54000/54000 [==============================] - 7s 138us/step - loss: 0.5798 - acc: 0.8107 - val_loss: 0.3855 - val_acc: 0.8807
Epoch 84/133
54000/54000 [==============================] - 7s 138us/step - loss: 0.6349 - acc: 0.8003 - val_loss: 0.2128 - val_acc: 0.9350
Epoch 85/133
54000/54000 [==============================] - 7s 135us/step - loss: 0.5422 - acc: 0.8246 - val_loss: 0.3271 - val_acc: 0.8968
Epoch 86/133
54000/54000 [==============================] - 9s 158us/step - loss: 0.5838 - acc: 0.8141 - val_loss: 0.2502 - val_acc: 0.9208
Epoch 87/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.5604 - acc: 0.8180 - val_loss: 0.2908 - val_acc: 0.9148
Epoch 88/133
54000/54000 [==============================] - 10s 177us/step - loss: 0.5759 - acc: 0.8129 - val_loss: 0.3673 - val_acc: 0.8818
Epoch 89/133
54000/54000 [==============================] - 7s 128us/step - loss: 0.6072 - acc: 0.8052 - val_loss: 0.2121 - val_acc: 0.9363
Epoch 90/133
54000/54000 [==============================] - 7s 131us/step - loss: 0.5465 - acc: 0.8238 - val_loss: 0.2058 - val_acc: 0.9368
Epoch 91/133
54000/54000 [==============================] - 9s 170us/step - loss: 0.5377 - acc: 0.8276 - val_loss: 0.5367 - val_acc: 0.8370
Epoch 92/133
54000/54000 [==============================] - 7s 139us/step - loss: 0.6635 - acc: 0.7921 - val_loss: 0.3039 - val_acc: 0.9072
Epoch 93/133
54000/54000 [==============================] - 7s 129us/step - loss: 0.5712 - acc: 0.8137 - val_loss: 0.2773 - val_acc: 0.9183
Epoch 94/133
54000/54000 [==============================] - 10s 181us/step - loss: 0.5605 - acc: 0.8197 - val_loss: 0.4018 - val_acc: 0.8797
Epoch 95/133
54000/54000 [==============================] - 9s 167us/step - loss: 0.6062 - acc: 0.8059 - val_loss: 0.2626 - val_acc: 0.9202
Epoch 96/133
54000/54000 [==============================] - 8s 146us/step - loss: 0.5763 - acc: 0.8172 - val_loss: 0.1947 - val_acc: 0.9417
Epoch 97/133
54000/54000 [==============================] - 8s 156us/step - loss: 0.5350 - acc: 0.8265 - val_loss: 0.2227 - val_acc: 0.9350
Epoch 98/133
54000/54000 [==============================] - 9s 172us/step - loss: 0.5465 - acc: 0.8258 - val_loss: 0.2052 - val_acc: 0.9357
Epoch 99/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.5324 - acc: 0.8280 - val_loss: 0.2124 - val_acc: 0.9393
Epoch 100/133
54000/54000 [==============================] - 9s 161us/step - loss: 0.5364 - acc: 0.8291 - val_loss: 0.1994 - val_acc: 0.9378
Epoch 101/133
54000/54000 [==============================] - 8s 157us/step - loss: 0.5339 - acc: 0.8271 - val_loss: 0.2096 - val_acc: 0.9382
Epoch 102/133
54000/54000 [==============================] - 8s 145us/step - loss: 0.5410 - acc: 0.8283 - val_loss: 0.2305 - val_acc: 0.9338
Epoch 103/133
54000/54000 [==============================] - 11s 204us/step - loss: 0.5383 - acc: 0.8256 - val_loss: 0.3488 - val_acc: 0.9008
Epoch 104/133
54000/54000 [==============================] - 14s 250us/step - loss: 0.5918 - acc: 0.8106 - val_loss: 0.3120 - val_acc: 0.9008
Epoch 105/133
54000/54000 [==============================] - 13s 238us/step - loss: 0.5821 - acc: 0.8159 - val_loss: 0.2411 - val_acc: 0.9292
Epoch 106/133
54000/54000 [==============================] - 13s 249us/step - loss: 0.5486 - acc: 0.8240 - val_loss: 0.2756 - val_acc: 0.9120
Epoch 107/133
54000/54000 [==============================] - 12s 216us/step - loss: 0.5584 - acc: 0.8200 - val_loss: 0.2550 - val_acc: 0.9177cc: 0
Epoch 108/133
54000/54000 [==============================] - 10s 192us/step - loss: 0.5602 - acc: 0.8202 - val_loss: 0.2627 - val_acc: 0.9170
Epoch 109/133
54000/54000 [==============================] - 13s 232us/step - loss: 0.5523 - acc: 0.8235 - val_loss: 0.2847 - val_acc: 0.9182
Epoch 110/133
54000/54000 [==============================] - 13s 246us/step - loss: 0.5672 - acc: 0.8174 - val_loss: 0.2042 - val_acc: 0.9362
Epoch 111/133
54000/54000 [==============================] - 10s 194us/step - loss: 0.5392 - acc: 0.8264 - val_loss: 0.2582 - val_acc: 0.9210
Epoch 112/133
54000/54000 [==============================] - 15s 280us/step - loss: 0.5570 - acc: 0.8216 - val_loss: 0.2599 - val_acc: 0.9210
Epoch 113/133
54000/54000 [==============================] - 13s 250us/step - loss: 0.5532 - acc: 0.8207 - val_loss: 0.2561 - val_acc: 0.9153
Epoch 114/133
54000/54000 [==============================] - 8s 157us/step - loss: 0.5579 - acc: 0.8212 - val_loss: 0.2083 - val_acc: 0.9332
Epoch 115/133
54000/54000 [==============================] - 8s 154us/step - loss: 0.5380 - acc: 0.8269 - val_loss: 0.3357 - val_acc: 0.9043
Epoch 116/133
54000/54000 [==============================] - 9s 174us/step - loss: 0.6166 - acc: 0.8102 - val_loss: 0.3330 - val_acc: 0.9023
Epoch 117/133
54000/54000 [==============================] - 12s 217us/step - loss: 0.5693 - acc: 0.8138 - val_loss: 0.2202 - val_acc: 0.9345
Epoch 118/133
54000/54000 [==============================] - 10s 182us/step - loss: 0.5379 - acc: 0.8271 - val_loss: 0.4368 - val_acc: 0.8708
Epoch 119/133
54000/54000 [==============================] - 10s 176us/step - loss: 0.6577 - acc: 0.7851 - val_loss: 0.3179 - val_acc: 0.9037
Epoch 120/133
54000/54000 [==============================] - 10s 178us/step - loss: 0.5965 - acc: 0.8123 - val_loss: 0.1930 - val_acc: 0.9412
Epoch 121/133
54000/54000 [==============================] - 7s 131us/step - loss: 0.5244 - acc: 0.8322 - val_loss: 0.4474 - val_acc: 0.8725
Epoch 122/133
54000/54000 [==============================] - 6s 117us/step - loss: 0.6087 - acc: 0.8103 - val_loss: 0.4378 - val_acc: 0.8718
Epoch 123/133
54000/54000 [==============================] - 9s 167us/step - loss: 0.6253 - acc: 0.7974 - val_loss: 0.3291 - val_acc: 0.9037
Epoch 124/133
54000/54000 [==============================] - 11s 197us/step - loss: 0.6002 - acc: 0.8144 - val_loss: 0.4941 - val_acc: 0.8537
Epoch 125/133
54000/54000 [==============================] - 8s 152us/step - loss: 0.6677 - acc: 0.7816 - val_loss: 0.3775 - val_acc: 0.8933
Epoch 126/133
54000/54000 [==============================] - 8s 144us/step - loss: 0.6198 - acc: 0.8007 - val_loss: 0.3835 - val_acc: 0.8918
Epoch 127/133
54000/54000 [==============================] - 8s 153us/step - loss: 0.6007 - acc: 0.8054 - val_loss: 0.9414 - val_acc: 0.7368
Epoch 128/133
54000/54000 [==============================] - 8s 141us/step - loss: 0.8110 - acc: 0.7734 - val_loss: 0.3031 - val_acc: 0.9087
Epoch 129/133
54000/54000 [==============================] - 8s 141us/step - loss: 0.5593 - acc: 0.8190 - val_loss: 0.3126 - val_acc: 0.9073
Epoch 130/133
54000/54000 [==============================] - 7s 133us/step - loss: 0.5919 - acc: 0.8179 - val_loss: 0.3175 - val_acc: 0.9030
Epoch 131/133
54000/54000 [==============================] - 9s 160us/step - loss: 0.5761 - acc: 0.8146 - val_loss: 0.4523 - val_acc: 0.8782
Epoch 132/133
54000/54000 [==============================] - 9s 161us/step - loss: 0.6465 - acc: 0.8079 - val_loss: 0.3022 - val_acc: 0.9115
Epoch 133/133
54000/54000 [==============================] - 8s 146us/step - loss: 0.5766 - acc: 0.8159 - val_loss: 0.2019 - val_acc: 0.9418

ポイントは、モデルを再現するために、同じシードを設定することです。再現しない場合は、仮にモデルの構造が同じでも、最適なepochを採用する意味が消失します(validation dataの最大精度が保証されないため)。

5. 本当の性能評価

最後に、テストデータを利用した場合の性能を求めてみます。これまで、validation dataの精度を見てきましたが、それは教師データから取り出された10%のデータのことでした。validation dataを使って良いパラメータを探してきましたが、逆に言えば、validation dataのみに適合するパラメータでしかありません。したがって、validation data以外、本当に最後まで一度も使わなかったデータで精度をチェックしてみます。

In [146]:
mm3_acc_test = mm3.evaluate(x_test_cnn, y_test, verbose=1, batch_size=32+1) # バッチサイズには2^n+1
print("epoch=200のモデルの正答率:")
print("・テストデータ: ", mm3_acc_test[1])

print("")
mm3best_acc_test = mm3_best.evaluate(x_test_cnn, y_test, verbose=1, batch_size=32+1) # バッチサイズには2^n+1
print("epoch=133のモデルの正答率:")
print("・テストデータ: ", mm3best_acc_test[1])
10000/10000 [==============================] - 3s 253us/step
epoch=200のモデルの正答率:
・テストデータ:  0.8995000250339508

10000/10000 [==============================] - 2s 190us/step
epoch=133のモデルの正答率:
・テストデータ:  0.9348000255942345

上が、epochを最適化していないモデル(epoch=2000)で、下がvalidation dataによりepochを最適化したモデル(epoch=133)です。これを見るとわかるように、epochを最適化した方が、最後まで一切手をつけなかったテストデータにおいても、性能が高いことがわかります。これは、validation dataに基づく最適化が、ある程度意味があることを示しています。

とはいえ、validation data とテストデータは、まったくの別物です。validation dataで最適化したからといって、テストデータでも良い性能が保証されるわけではありません。あくまで、その傾向があるだけです。

いくつかのtips

Q. 一度プログラムを閉じても、今回の方法だと、まったく同じモデルになるのか?

なります。たとえば、1月1日にモデルを作って、プログラムを閉じて、2月1日にもう一回モデルを作ったとします。このとき、乱数シードを正しく設定していれば、まったく同じモデルが出来上がります。ただ、バージョンを変えたりすると、どうなるかわかりません。

Q. accとval_acc、なんでval_accの方が高い? 普通に考えたら、accの方が高いのでは?

accは学習時の正答率です。ここで、レイヤの間にdropoutをつけた場合、そのレイヤの出力値が確率に依存して0に書き換えられます。これは過学習の抑止になりますが、学習時の「見かけ上の」正答率は下がります。本当はdropoutされない正答率を表示して欲しいところですが、残念ながら、そうなっていません。したがって、accが低い値を示します。一方、val_accは学習に使用しないデータの正答率ですから、dropoutにより出力値が書き換えられていません。このため、accが低く出て、val_accが高く出るわけです。モデル構造のところで、dropoutをコメントアウトすると、accがval_accよりも大幅に低いという現象がなくなることを確認できると思います。

Q. 途中から学習を再開したい。

50回学習させた後に、やっぱり100回くらい学習させればよかったなどと思うことがよくあります。この場合、最初から学習を開始すると、時間が勿体無いと思うことがあります。このときは、SequentialやCompileを経由させずに、もう一度fit関数を実行すると、途中から学習を開始してくれます(でも、これをすると何回学習したのかわからなくなる場合があります、あくまで試行錯誤時のテクニックと捉えてください)。

Q. 不均衡データの対応方法は?

クラス0とクラス1の2択問題を考えます。このとき、クラス0が4500データ、クラス1が500データしかなかったとします。明らかにクラス1の方が少ないことになります。この場合、すべてクラス0と判断するダメなモデルがあったとしても、正答率が90%となってしまい、見かけ上うまくいっているように見えてしまいます。また、学習もうまくいかないことが多いです(クラス1と出力するパラメータを獲得しようとしても、結果的に正答率が減少してしまうことが多いため、適切な学習が行われにくい)。このようなデータを不均衡データと呼びます。この場合は、

  • cw0 = 1
  • cw1 = 8
  • cw= {0:cw0, 1:cw1}
  • model.fit(X, Y, class_weight=cw)

のように、fit関数のclass_weight引数に、不均衡を解消するような重みを与えます。上の例では、学習時、クラス1の正答に対して8倍のボーナスがかかります。これで、学習が適切に行われる場合があります。ただし、これはあくまでソフト上の改善案であって、たとえ集めにくいという事情があったとしても、本来はクラス1のデータを集めることが真っ当な解決方法ですから、それを心に留めておきましょう。なお、この処理で表示されるaccやval_accなどは、重みづいたものではないので、注意してください。epochを重ねても正答率が90%のまま推移しますが、それをみても、すべてクラス0と回答することによる90%か、クラス1と回答する場合があった上での90%か、みてもわかりません。学習後、実際に自分の目で一つ一つの判定結果を見る必要があります。