Tensor

grad_fn

Each variable has a .grad_fn attribute that references a function that has created a function (except for Tensors created by the user, these have None as .grad_fn).

For example, we create x=torch.ones(2,2, requires_grad=True), then x.grad_fn = None, since we create the tensor x ourselves. And for y=x+2, then y is tensor([[3,3]], grad_fn=<AddBackward0>).

When run y_pred = model(X_test_tensor), get the following error RuntimeError: Expected object of scalar type Float but got scalar type Double for xxxxx. Solve by y_pred = model(X_test_tensor.float()).

Debug CUDA

• In command line, use CUDA_LAUNCH_BLOCKING=1 python <script>.py

Watch GPU usage

• watch -n 0.5 nvidia-smi

Multi-gpu, results is a list

item 0 in the list, is results assigned to gpu 0 1, 1. etc