Nrthugu

Trying to serve a Tensorflow tensor2tensor Chinese to English model and am having trouble querying my model. This is the client error I am receiving:

(test) root@ubuntu-c-8-16gib-sfo2-01:~/T2T_Model# t2t-query-server   --server=0.0.0.0:9000   --servable_name=transformer   --problem=translate_enzh_wmt32k_rev   --data_dir=/root/T2T_Model/t2t_data   --inputs_once='Hello my name is John.'



Traceback (most recent call last):

  File "/usr/local/bin/t2t-query-server", line 17, in <module>

    tf.app.run()

  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run

    _sys.exit(main(argv))

  File "/usr/local/bin/t2t-query-server", line 12, in main

    query.main(argv)

  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/query.py", line 89, in main

    outputs = serving_utils.predict([inputs], problem, request_fn)

  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/serving_utils.py", line 157, in predict

    predictions = request_fn(examples)

  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/serving_utils.py", line 113, in _make_grpc_request

    response = stub.Predict(request, timeout_secs)

  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 533, in __call__

    return _end_unary_response_blocking(state, call, False, None)

  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking

    raise _Rendezvous(state, None, None, deadline)

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:

    status = StatusCode.INVALID_ARGUMENT

    details = "Requested more than 0 entries, but params is empty.  Params shape: [1,4,8,0,64]

     [[{{node transformer/while/GatherNd_32}} = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _output_shapes=[[?,8,?,?,64]], _device="/job:localhost/replica:0/task:0/device:CPU:0"](transformer/while/Reshape_65, transformer/while/stack)]]"

    debug_error_string = "{"created":"@1542086942.107507941","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Requested more than 0 entries, but params is empty.  Params shape: [1,4,8,0,64]nt [[{{node transformer/while/GatherNd_32}} = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _output_shapes=[[?,8,?,?,64]], _device="/job:localhost/replica:0/task:0/device:CPU:0"](transformer/while/Reshape_65, transformer/while/stack)]]","grpc_status":3}"

>

The model server seems to be working fine and responding with the same error:

(test) root@ubuntu-c-8-16gib-sfo2-01:~/T2T_Model# tensorflow_model_server   --port=9000   --model_name=transformer   --model_base_path=/root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export

2018-11-13 05:28:29.116290: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config:  model_name: transformer model_base_path: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export

2018-11-13 05:28:29.116412: I tensorflow_serving/model_servers/server_core.cc:461] Adding/updating models.

2018-11-13 05:28:29.116424: I tensorflow_serving/model_servers/server_core.cc:558]  (Re-)adding model: transformer

2018-11-13 05:28:29.216782: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: transformer version: 1542073770}

2018-11-13 05:28:29.216806: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: transformer version: 1542073770}

2018-11-13 05:28:29.216815: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: transformer version: 1542073770}

2018-11-13 05:28:29.216830: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770

2018-11-13 05:28:29.216838: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770

2018-11-13 05:28:29.537966: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }

2018-11-13 05:28:29.597214: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA

2018-11-13 05:28:29.722289: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:162] Restoring SavedModel bundle.

2018-11-13 05:28:30.139345: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:138] Running MainOp with key saved_model_main_op on SavedModel bundle.

2018-11-13 05:28:30.227063: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags { serve }; Status: success. Took 1010210 microseconds.

2018-11-13 05:28:30.227116: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770/assets.extra/tf_serving_warmup_requests

2018-11-13 05:28:30.227223: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: transformer version: 1542073770}

2018-11-13 05:28:30.229398: I tensorflow_serving/model_servers/server.cc:286] Running gRPC ModelServer at 0.0.0.0:9000 ...



2018-11-13 05:59:38.052592: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: Requested more than 0 entries, but params is empty.  Params shape: [1,4,8,0,64]

Would appreciate any comments or suggestions as to what is wrong here.