Nrthugu

I was following the official TF giude to use the tf.data.Dataset API for building data pipeline, but I found it ~2 times slower than my python data pipeline using hdf5.

Here's my experiment result:

TFRecordDataset: 660 thousand samples/second 

hdf5 reader + feed_dict to placeholder: 1.1 million samples/second

My experiment setting is:

- batch size=1000,

- print log every 10 million samples to show the time,

- run a fake model that only takes data as input and do not compute anything.

what's worse is that when I set batch size=10000, it becomes:

TFRecordDataset: 760 thousand samples/second 

hdf5 reader + feed_dict to placeholder: 2.1 million samples/second

Here's my code for reading tfrecord

def parse_tfrecord_batch(record_batch):

    FEATURE_LEN = 29

    dics = {

        'label': tf.FixedLenFeature(shape=(), dtype=tf.int64),

        'feature_id': tf.FixedLenFeature(shape=(FEATURE_LEN,), dtype=tf.int64),

        'feature_val': tf.FixedLenFeature(shape=(FEATURE_LEN,), dtype=tf.float32)

    }

    parsed_example = tf.parse_example(record_batch, dics)

    return parsed_example['feature_id'], parsed_example['feature_val'], parsed_example['label']



class fakeModel:

    def __init__(self, train_filenames):

        self.graph = tf.Graph()

        with self.graph.as_default():

            dataset = tf.data.TFRecordDataset(train_filenames)

            dataset = dataset.repeat()

            dataset = dataset.batch(1000)

            dataset = dataset.map(parse_tfrecord_batch, num_parallel_calls=1)

            dataset = dataset.prefetch(1000)

            self.iterator = dataset.make_initializable_iterator()

            self.id, self.wt, self.label = self.iterator.get_next()



            self.train_preds = tf.identity(self.lbl_hldr)

I've tune the num_parallel_calls to 2, 10. Not working.

I've also tuned the prefetch(n) from 1 to 1000, which has little improvement.

My question is:

Is there any way to improve my tfrecord data pipeline? Am I missing something in my code?

Appreciate it for any help.