title: Read formatted data from part of a file fast (Gmsh mesh format)
body: <p>I maintain <a href="https://github.com/nschloe/meshio" rel="noreferrer">a little Python package</a> that converts between different formats used for mesh representation à la</p>
  
  <p><a href="https://i.stack.imgur.com/yuqfZ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/yuqfZ.png" alt="enter image description here"></a></p>
  
  <p>Those files can grow pretty big, so when reading them with Python it's important to do it efficiently.</p>
  
  <p>One of the most used formats is <a href="http://www.manpagez.com/info/gmsh/gmsh-2.2.6/gmsh_63.php" rel="noreferrer">msh</a> from <a href="http://gmsh.info/" rel="noreferrer">Gmsh</a>. Unfortunately, its data layout is arguably not the best. An example file:</p>
  
  
  
  <pre><code>$MeshFormat
  2.2 0 8
  $EndMeshFormat
  $Nodes
  8
  1 -0.5 -0.5 -0.5
  2  0.5 -0.5 -0.5
  3 -0.5  0.5 -0.5
  4  0.5  0.5 -0.5
  5 -0.5 -0.5  0.5
  6  0.5 -0.5  0.5
  7 -0.5  0.5  0.5
  8  0.5  0.5  0.5
  $EndNodes
  $Elements
  2
  1 4 2 1 11 1 2 3 5
  2 4 2 1 11 2 5 6 8
  $EndElements
  </code></pre>
  
  <ul>
  <li><p>For the <code>$Nodes</code>:</p>
  
  <p>The first number (<code>8</code>) is the number of nodes to follow.</p>
  
  <p>In each node line, the first number is the <strong>index</strong> (not actually needed by still part of the format, ugh), then follow the <strong>three spatial coordinates</strong>.</p>
  
  <p>So far, I haven't come up with anything better than <code>islice</code>s in a  <code>for</code>  loop, which is pretty slow.</p></li>
  </ul>
  
  
  
  <pre><code># The first line is the number of nodes
  line = next(islice(f, 1))
  num_nodes = int(line)
  #
  points = numpy.empty((num_nodes, 3))
  for k, line in enumerate(islice(f, num_nodes)):
      points[k, :] = numpy.array(line.split(), dtype=float)[1:]
      line = next(islice(f, 1))
  assert line.strip() == '$EndNodes'
  </code></pre>
  
  <ul>
  <li><p>For the <code>$Elements</code>:</p>
  
  <p>The first number (<code>2</code>) is the number of elements to follow.</p>
  
  <p>In each element line, the first number is the <strong>index</strong>, then follows an enum for the <strong>element type</strong> (<code>4</code> is for tetrahedra). Then follows the <strong>number of integer tags</strong> for this element (<code>2</code> in each case here, namely <code>1</code> and <code>11</code>).  Corresponding to the element type, the last few entries in this row correspond to <strong><code>$Node</code> indices</strong> that form the element – in the case of a tetrahedron, the last four entries.</p>
  
  <p>Since the number of tags can vary from element to element (i.e., line to line), just like the element type and the number of node indices, each line may have a different number of integers.</p></li>
  </ul>
  
  <p>For both <code>$Nodes</code> and <code>$Elements</code>, any help for reading this data quickly is appreciated.</p>