[keras]UserWarning: The input 10 could not be retrieved. It could be because a worker has died. UserWarning)

environment

• Colab, hardware accelerator : GPU
• python : 3.6.8
• tensorflow : 1.13.1

Error

/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 10 could not be retrieved. It could be because a worker has died.

Situation

Google Colab에서 Efficientnet(Ref3) 모델을 학습하는 도중 위와 같은 log가 출력되었다.
Ref1에 따르면 별 문제가 없다고 하지만 1 Epoch이 끝난 다음 두 번째 Epoch이 절대 시작되지 않았다

 

계속 찾아본 결과 똑같은 상황을 겪고 있는 Ref2를 발견하였다
여기서 제시하는 문제점으로는 1. 멀티 프로세싱 모드 사용 2. 구글 드라이브 마운트 상태 가 있었다
그래서 use_multiprocessing=True와 workers=0 부분을 없애고 나니 학습이 매우 잘 된다.

Code

import datetime
import tensorflow as tf
from keras.applications import Xception
from keras.utils import multi_gpu_model
start = datetime.datetime.now()

num_classes = 2
with tf.device('/gpu:0'):
  model = Xception(weights=None,
                   input_shape=(height, width, 3),
                   classes=num_classes)

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])
history = model.fit_generator(
      train_generator,
      steps_per_epoch= NUM_TRAIN //batch_size,
      epochs=epochs,
      validation_data=validation_generator,
      validation_steps= NUM_TEST //batch_size,
      verbose=1
      use_multiprocessing=True,
      workers=0)

end = datetime.datetime.now()
time_delta = end - start

Ref

1. https://stackoverflow.com/questions/58446290/userwarning-an-input-could-not-be-retrieved-it-could-be-because-a-worker-has
2. https://datascience.stackexchange.com/questions/60647/keras-userwarning-the-input-1303-could-not-be-retrieved-it-could-be-because-a
3. https://github.com/Tony607/efficientnet_keras_transfer_learning