Help! training not starting! #urgent

[danialvi]

My training is not starting. I have used python 3.6 with tensorflow gpu 1.8.0 and keras 2.1.2. Also I have a Geforce GTX 3060 running on my computer. So it shouldnt be a problem. I also installed Norton antivirus on this new computer. On the older computer which has a bad GPU I had Panda Dome, but there training was running. But after over 1 hour, the training was only on 1%. Thats why I bought a new computer with a good GPU and CPU. Some of this work is going to be presented in my master thesis. I would appreciate any help soon.

[danialvi]

I got this error now:

---

InternalError Traceback (most recent call last)
C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1321 try:
-> 1322 return fn(*args)
1323 except errors.OpError as e:

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
1306 return self._call_tf_sessionrun(
-> 1307 options, feed_dict, fetch_list, target_list, run_metadata)
1308

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
1408 self._session, options, feed_dict, fetch_list, target_list,
-> 1409 run_metadata)
1410 else:

InternalError: Blas GEMM launch failed : a.shape=(30, 64), b.shape=(64, 10), m=30, n=10, k=64
[[Node: dense2/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@training/Nadam/gradients/dropout_2/cond/Merge_grad/cond_grad"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dropout_2/cond/Merge, dense2/kernel/read)]]
[[Node: loss/mul/_129 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1107_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

InternalError Traceback (most recent call last)
in
1 history = model.fit_generator(train_generator, steps_per_epoch=num_train_examples//batch_size, epochs=500, callbacks=callbacks,
----> 2 validation_data=eval_generator, validation_steps=num_eval_examples//batch_size, verbose=2)

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2145 outs = self.train_on_batch(x, y,
2146 sample_weight=sample_weight,
-> 2147 class_weight=class_weight)
2148
2149 if not isinstance(outs, list):

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
1837 ins = x + y + sample_weights
1838 self._make_train_function()
-> 1839 outputs = self.train_function(ins)
1840 if len(outputs) == 1:
1841 return outputs[0]

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\backend\tensorflow_backend.py in call(self, inputs)
2355 session = get_session()
2356 updated = session.run(fetches=fetches, feed_dict=feed_dict,
-> 2357 **self.session_kwargs)
2358 return updated[:len(self.outputs)]
2359

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
898 try:
899 result = self._run(None, fetches, feed_dict, options_ptr,
--> 900 run_metadata_ptr)
901 if run_metadata:
902 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1133 if final_fetches or final_targets or (handle and feed_dict_tensor):
1134 results = self._do_run(handle, final_targets, final_fetches,
-> 1135 feed_dict_tensor, options, run_metadata)
1136 else:
1137 results = []

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1314 if handle is None:
1315 return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1316 run_metadata)
1317 else:
1318 return self._do_call(_prun_fn, handle, feeds, fetches)

C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1333 except KeyError:
1334 pass
-> 1335 raise type(e)(node_def, op, message)
1336
1337 def _extend_graph(self):

InternalError: Blas GEMM launch failed : a.shape=(30, 64), b.shape=(64, 10), m=30, n=10, k=64
[[Node: dense2/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@training/Nadam/gradients/dropout_2/cond/Merge_grad/cond_grad"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dropout_2/cond/Merge, dense2/kernel/read)]]
[[Node: loss/mul/_129 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1107_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'dense2/MatMul', defined at:
File "C:\ProgramData\anaconda3\envs\airsim\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\ProgramData\anaconda3\envs\airsim\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\traitlets\config\application.py", line 664, in launch_instance
app.start()
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\kernelapp.py", line 612, in start
self.io_loop.start()
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\platform\asyncio.py", line 199, in start
self.asyncio_loop.run_forever()
File "C:\ProgramData\anaconda3\envs\airsim\lib\asyncio\base_events.py", line 442, in run_forever
self._run_once()
File "C:\ProgramData\anaconda3\envs\airsim\lib\asyncio\base_events.py", line 1462, in _run_once
handle._run()
File "C:\ProgramData\anaconda3\envs\airsim\lib\asyncio\events.py", line 145, in _run
self._callback(*self._args)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\ioloop.py", line 688, in
lambda f: self._run_callback(functools.partial(callback, future))
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\ioloop.py", line 741, in _run_callback
ret = callback()
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 814, in inner
self.ctx_run(self.run)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 162, in _fake_ctx_run
return f(*args, **kw)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 775, in run
yielded = self.gen.send(value)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one
yield gen.maybe_future(dispatch(*args))
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 234, in wrapper
yielded = ctx_run(next, result)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 162, in _fake_ctx_run
return f(*args, **kw)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 234, in wrapper
yielded = ctx_run(next, result)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 162, in _fake_ctx_run
return f(*args, **kw)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\kernelbase.py", line 545, in execute_request
user_expressions, allow_stdin,
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 234, in wrapper
yielded = ctx_run(next, result)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tornado\gen.py", line 162, in _fake_ctx_run
return f(*args, **kw)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\ipkernel.py", line 306, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\interactiveshell.py", line 2867, in run_cell
raw_cell, store_history, silent, shell_futures)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\interactiveshell.py", line 2895, in _run_cell
return runner(coro)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\async_helpers.py", line 68, in pseudo_sync_runner
coro.send(None)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\interactiveshell.py", line 3072, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes
if (await self.run_code(code, result, async=asy)):
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 24, in
merged = Dense(10, activation=activation, name='dense2')(merged)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\engine\topology.py", line 603, in call
output = self.call(inputs, **kwargs)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\layers\core.py", line 843, in call
output = K.dot(inputs, self.kernel)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\keras\backend\tensorflow_backend.py", line 1057, in dot
out = tf.matmul(x, y)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2122, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 4278, in mat_mul
name=name)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\ProgramData\anaconda3\envs\airsim\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(30, 64), b.shape=(64, 10), m=30, n=10, k=64
[[Node: dense2/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@training/Nadam/gradients/dropout_2/cond/Merge_grad/cond_grad"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dropout_2/cond/Merge, dense2/kernel/read)]]
[[Node: loss/mul/_129 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1107_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[danialvi]

@mitchellspryn please help

[danialvi]

@adshar

[danialvi]

depencies.txt
Here is the list of dependencies I have in my anaconda env:

[mitchellspryn]

I am not at MSFT currently, so I am not actively supporting this repo any more.

That said, I took a look at your stack trace. It looks like CUDA isn't installed properly. Relevant portion:

InternalError: Blas GEMM launch failed : a.shape=(30, 64), b.shape=(64, 10), m=30, n=10, k=64
[[Node: dense2/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@training/Nadam/gradients/dropout_2/cond/Merge_grad/cond_grad"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dropout_2/cond/Merge, dense2/kernel/read)]]

I'd check to see if you can run any keras training operation - e.g. try training a linear model on some random data points and see if the forward/backpropagation works properly. My guess is no, and that'll help you debug what the situation is with your cuda install.

[danialvi]

I am not at MSFT currently, so I am not actively supporting this repo any more.

That said, I took a look at your stack trace. It looks like CUDA isn't installed properly. Relevant portion:

InternalError: Blas GEMM launch failed : a.shape=(30, 64), b.shape=(64, 10), m=30, n=10, k=64
[[Node: dense2/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@training/Nadam/gradients/dropout_2/cond/Merge_grad/cond_grad"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dropout_2/cond/Merge, dense2/kernel/read)]]

I'd check to see if you can run any keras training operation - e.g. try training a linear model on some random data points and see if the forward/backpropagation works properly. My guess is no, and that'll help you debug what the situation is with your cuda install.

Thank you for answering. I have tried to reinstall to check if it's something to do with cuda. I also tried by installing the cudatoolkit and cudann before install tensorflow by following these steps:
conda install cudatoolkit=9.0
conda install cudnn=7.1.4=cuda9.0_0
conda install -c anaconda tensorflow-gpu=1.8.0
conda install -c anaconda keras-gpu=2.1.2
python -m pip install --upgrade pip
conda update -n base conda
pip install msgpack-rpc-python
pip uninstall tornado
conda install -c conda-forge tornado=4.5.3
conda install jupyter
pip install matplotlib==2.1.2
pip install image
pip install keras_tqdm
conda install -c conda-forge opencv
conda install pandas
pip install --upgrade numpy==1.16.4
conda install scipy
pip install opencv-python
pip install --upgrade h5py==2.10.0
python -m ipykernel install --user

Still I have the same problem. Do you have any idea how I can solve this? I have really tried to look it up, but it seems many had the same problem, but no solutions that worked for me. As I am using this as a part of my master thesis, I have limited time as well.