<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">Currently, the <code class="notranslate">DataParallel</code>-based approach for distributed training does not support complex tensors. Some collective Operations (operations involving communication between multiple nodes in a distributed training setup) that are a part of the <code class="notranslate">torch.distributed</code> package are currently in the process of supporting complex-valued tensors. Since underlying collective libraries such as NCCL do not natively support complex types, we instead use constant-time views, such as <code class="notranslate">torch.view_as_real</code> and <code class="notranslate">torch.view_as_complex</code>.</p>
<p dir="auto">Since the <code class="notranslate">DataParallel</code> package uses a collective called <code class="notranslate">broadcast_coalesced</code> in order to duplicate the model across nodes, complex tensor support must be added to <code class="notranslate">broadcast_coalesced</code> in order to enable distributed training with complex tensors using <code class="notranslate">DataParallel</code>.</p>
<h2 dir="auto">References</h2>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="713837134" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/45760" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/45760/hovercard" href="https://github.com/pytorch/pytorch/issues/45760">#45760</a>: Umbrella Issue for Complex Tensor Support<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="727779220" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/46745" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/46745/hovercard" href="https://github.com/pytorch/pytorch/issues/46745">#46745</a>: User Request for Complex Tensor Support in DataParallel<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="715242595" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/45879" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/45879/hovercard" href="https://github.com/pytorch/pytorch/pull/45879">#45879</a>: Example PR that added Complex Tensor Support to some collectives</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jiayisuse/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jiayisuse">@jiayisuse</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/agolynski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/agolynski">@agolynski</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions and Help</h2>
<p dir="auto">Does JIT trace support the model with MKLDNN layout?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torchvision
from torch.utils import mkldnn as mkldnn_utils

# An instance of your model.
model = torchvision.models.resnet18(pretrained=True)
# Switch the model to eval model
model.eval()

model = mkldnn_utils.to_mkldnn(model)

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224).to_mkldnn()

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

# Save the TorchScript model
traced_script_module.save(&quot;traced_resnet_model.pt&quot;)"><pre class="notranslate"><code class="notranslate">import torch
import torchvision
from torch.utils import mkldnn as mkldnn_utils

# An instance of your model.
model = torchvision.models.resnet18(pretrained=True)
# Switch the model to eval model
model.eval()

model = mkldnn_utils.to_mkldnn(model)

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224).to_mkldnn()

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

# Save the TorchScript model
traced_script_module.save("traced_resnet_model.pt")
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;pytorch.py&quot;, line 17, in &lt;module&gt;
    traced_script_module = torch.jit.trace(model, example)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py&quot;, line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py&quot;, line 1119, in trace_module
    check_tolerance, strict, _force_outplace, True, _module_class)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py&quot;, line 15, in decorate_context
    return func(*args, **kwargs)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py&quot;, line 597, in _check_trace
    copied_dict[name] = _clone_inputs(data)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py&quot;, line 450, in _clone_inputs
    clone_input, condition_msg=&quot;tensors&quot;)(args)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/autograd/function.py&quot;, line 263, in _map
    return fn(obj)
  File &quot;/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py&quot;, line 443, in clone_input
    v = a.detach().clone(memory_format=torch.preserve_format).requires_grad_(a.requires_grad)
RuntimeError: unsupported memory format option Preserve"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "pytorch.py", line 17, in &lt;module&gt;
    traced_script_module = torch.jit.trace(model, example)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py", line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py", line 1119, in trace_module
    check_tolerance, strict, _force_outplace, True, _module_class)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py", line 597, in _check_trace
    copied_dict[name] = _clone_inputs(data)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py", line 450, in _clone_inputs
    clone_input, condition_msg="tensors")(args)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/autograd/function.py", line 263, in _map
    return fn(obj)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/jit/__init__.py", line 443, in clone_input
    v = a.detach().clone(memory_format=torch.preserve_format).requires_grad_(a.requires_grad)
RuntimeError: unsupported memory format option Preserve
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/suo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/suo">@suo</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gmagogsfm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gmagogsfm">@gmagogsfm</a></p>