How To Get Data Generator More Efficient?

To train a neural network, I modified a code I found on YouTube. It looks as follows: def data_generator(samples, batch_size, shuffle_data = True, resize=224): num_samples = len(

Solution 1:

I have found your question very intriguing because you give only clues. So here is my investigation.

Using your snippets, I have found GitHub repository and 3 part video tutorial on YouTube that mainly focuses on the benefits of using generator functions in Python. The data is based on this kaggle (I would recommend to check out different kernels on that problem to compare the approach that you already tried with another CNN networks and review API in use).

You do not need to write a data generator from scratch, though it is not hard, but inventing the wheel is not productive.

• Keras has the ImageDataGenerator class.
• Plus here is a more generic example for DataGenerator.
• Tensorflow offers very neat pipelines with their tf.data.Dataset.

Nevertheless, to solve the kaggle's task, the model needs to perceive single images only, hence the model is a simple deep CNN. But as I understand, you are combining 8 random characters (classes) into one image to recognize multiple classes at once. For that task, you need R-CNN or YOLO as your model. I just recently opened for myself YOLO v4, and it is possible to make it work for specific task really quick.

General advice about your design and code.

• Make sure the library uses GPU. It saves a lot of time. (Even though I repeated flowers experiment from the repository very fast on CPU - about 10 minutes, but resulting predictions are no better than a random guess. So full training requires a lot of time on CPU.)
• Compare different versions to find a bottleneck. Try a dataset with 48 images (1 per class), increase the number of images per class, and compare. Reduce image size, change the model structure, etc.
• Test brand new models on small, artificial data to prove the idea or use iterative process, start from projects that can be converted to your task (handwriting recognition?).