title: Openstack: Horizon - "Unable to retrieve usage information" error after VM Shutdown
body: <p>I have setup Openstack (Newton) with Nova (Compute) on Ubuntu 16.04.1 LTS Desktop as a virtual machine guest (Oracle VM VirtualBox ver.5.1.8) on Windows 10 host.</p>
  
  <p>The installation was successful and could access the Horizon after the login process via <a href="http://192.168.56.102/dashboard/" rel="nofollow noreferrer">http://192.168.56.102/dashboard/</a> (from the Host) as shown in the figure below. (Can also access via <a href="http://10.0.2.15/dashboard/" rel="nofollow noreferrer">http://10.0.2.15/dashboard/</a> from the Guest but not shown here)</p>
  
  <p><a href="https://i.stack.imgur.com/Tynd4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Tynd4.png" alt="enter image description here"></a></p>
  
  <p>When the VM was shut down and later switched on again, the access to Openstack using Horizon’s dashboard was lost. “Error: Unable to retrieve usage information” is shown on the dashboard (as in the top right position of the figure below). This issue is a hard problem and could be replicated all the time.</p>
  
  <p><a href="https://i.stack.imgur.com/uaAo8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uaAo8.png" alt="enter image description here"></a></p>
  
  <p>The below is the snipped of error shown in /var/log/apache2/horizon_error.log file when Computer->Overview option was selected from the Horizon’s dashboard after the login process (that produces Figure 2 above).</p>
  
  <pre><code>2016-12-02 13:03:08.786092 Recoverable error: HTTPConnectionPool(host='10.0.2.15', port=8774): Max retries exceeded with url: /v2.1/extensions (Caused by NewConnectionError('&lt;requests.packages.urllib3.connection.HTTPConnection object at 0x7f3436ebeb10&gt;: Failed to establish a new connection: [Errno 111] Connection refused',))
  2016-12-02 13:03:08.804729 DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://10.0.2.15/identity/v3/users/9af3721f2d8e41cc9342b9941bafc5d1/projects -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}67ffd6b14a7e8891ae7cc8b29a639b81822c7b45"
  2016-12-02 13:03:08.935723 DEBUG:keystoneauth.session:RESP: [200] Date: Fri, 02 Dec 2016 13:03:08 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token x-openstack-request-id: req-fa398fe1-c5c9-4ce6-986b-cec266278f04 Content-Length: 966 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json 
  2016-12-02 13:03:08.935791 RESP BODY: {"links": {"self": "http://10.0.2.15/identity/v3/users/9af3721f2d8e41cc9342b9941bafc5d1/projects", "previous": null, "next": null}, "projects": [{"is_domain": false, "description": "Bootstrap project for initializing the cloud.", "links": {"self": "http://10.0.2.15/identity/v3/projects/4350966521b54001aeb666b6e55b7deb"}, "enabled": true, "id": "4350966521b54001aeb666b6e55b7deb", "parent_id": "default", "domain_id": "default", "name": "admin"}, {"is_domain": false, "description": "", "links": {"self": "http://10.0.2.15/identity/v3/projects/43dba94541ba4e8e83d5528038346ea9"}, "enabled": true, "id": "43dba94541ba4e8e83d5528038346ea9", "parent_id": "default", "domain_id": "default", "name": "demo"}, {"is_domain": false, "description": "", "links": {"self": "http://10.0.2.15/identity/v3/projects/be3ee2f8cb074628ac24c8be6e4c7160"}, "enabled": true, "id": "be3ee2f8cb074628ac24c8be6e4c7160", "parent_id": "default", "domain_id": "default", "name": "alt_demo"}]}
  2016-12-02 13:03:08.935809 
  2016-12-02 13:03:09.018515 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Load Balancers panel will not be displayed.
  2016-12-02 13:03:09.024775 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Firewalls panel will not be displayed.
  2016-12-02 13:03:09.029002 Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. VPN panel will not be displayed.
  2016-12-02 13:03:09.035255 Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will not be displayed
  </code></pre>
  
  <p>As the error about indicated problem in communicating with the endpoint. Verified if the endpoint for the services is ok in the VM, unfortunately it is not.</p>
  
  <pre><code>svanan@svanan:~/devstack$ openstack endpoint list
  Password: 
  Missing parameter(s): 
  Set a username with --os-username, OS_USERNAME, or auth.username
  Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url
  </code></pre>
  
  <p>The below is the original output in the VM when before it was shutdown.</p>
  
  <pre><code>svanan@svanan:~/devstack$ openstack endpoint list
  +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
  | ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                     |
  +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
  | 07480eca2071416c8dec30bafa95ff8f | RegionOne | neutron      | network        | True    | admin     | http://10.0.2.15:9696/                  |
  | 0aa32dc14c194561816b6d1b60d04404 | RegionOne | glance       | image          | True    | admin     | http://10.0.2.15:9292                   |
  | 0e5365a8ab3744c780feb25ecffc2c93 | RegionOne | keystone     | identity       | True    | internal  | http://10.0.2.15/identity               |
  | 15d8608b021f42e2a3b0402889a9064f | RegionOne | neutron      | network        | True    | public    | http://10.0.2.15:9696/                  |
  | 289a99bb83e846b2bfc5eabfb800d86d | RegionOne | glance       | image          | True    | internal  | http://10.0.2.15:9292                   |
  | 41397886443d44b7b6d857fccb34d55d | RegionOne | neutron      | network        | True    | internal  | http://10.0.2.15:9696/                  |
  | 4458eed550ae44479feca25f9bec96ce | RegionOne | nova_legacy  | compute_legacy | True    | public    | http://10.0.2.15:8774/v2/$(project_id)s |
  | 51be7c90e6c4437fbb75c3a40857774f | RegionOne | keystone     | identity       | True    | public    | http://10.0.2.15/identity               |
  | 7526c5a7e5084a07b1d1c7affe261ae4 | RegionOne | nova_legacy  | compute_legacy | True    | admin     | http://10.0.2.15:8774/v2/$(project_id)s |
  | 7c1e026c1c2d4f0b9dfdfe15860d6c5b | RegionOne | keystone     | identity       | True    | admin     | http://10.0.2.15/identity_v2_admin      |
  | 8641d88381fc499688162e8bcde5456c | RegionOne | cinderv2     | volumev2       | True    | internal  | http://10.0.2.15:8776/v2/$(project_id)s |
  | 8a194a2954a943568ccaf8bce5eb9400 | RegionOne | cinder       | volume         | True    | internal  | http://10.0.2.15:8776/v1/$(project_id)s |
  | 8c0b4d932da34c5990e43c0877146819 | RegionOne | nova         | compute        | True    | admin     | http://10.0.2.15:8774/v2.1              |
  | 94e8cae799954ba09abcd3b31b1c926e | RegionOne | nova         | compute        | True    | public    | http://10.0.2.15:8774/v2.1              |
  | a1527968e91d4df9b98fe09039e80bed | RegionOne | cinderv2     | volumev2       | True    | public    | http://10.0.2.15:8776/v2/$(project_id)s |
  | a3e47c8f82d546c8917a8223b8b81a74 | RegionOne | cinderv3     | volumev3       | True    | internal  | http://10.0.2.15:8776/v3/$(project_id)s |
  | b39a19918016462db99cf1de151fc870 | RegionOne | cinderv3     | volumev3       | True    | public    | http://10.0.2.15:8776/v3/$(project_id)s |
  | d17514aa0eec4752848df9599dd8cd00 | RegionOne | cinder       | volume         | True    | admin     | http://10.0.2.15:8776/v1/$(project_id)s |
  | db570fe7343b461a95cdd4179e1b28d9 | RegionOne | glance       | image          | True    | public    | http://10.0.2.15:9292                   |
  | eb4aa71c03b0418eadfed00715744b07 | RegionOne | cinderv3     | volumev3       | True    | admin     | http://10.0.2.15:8776/v3/$(project_id)s |
  | ee474205e562411abc1842dc9bccbb5d | RegionOne | nova_legacy  | compute_legacy | True    | internal  | http://10.0.2.15:8774/v2/$(project_id)s |
  | f383e0696ee647f79f6fd3a3d7ac5dff | RegionOne | cinder       | volume         | True    | public    | http://10.0.2.15:8776/v1/$(project_id)s |
  | f4239a1f86144d8dab3ed94237d7c10f | RegionOne | cinderv2     | volumev2       | True    | admin     | http://10.0.2.15:8776/v2/$(project_id)s |
  | fe679fcbc79e4664b7897a381e90dd78 | RegionOne | nova         | compute        | True    | internal  | http://10.0.2.15:8774/v2.1              |
  +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------+
  </code></pre>
  
  <p>Verified ifconfig at VM, it has the enp0s3, enp0s8, lo, virbr0  interfaces configured except the br-ex which was there before the shutdown as below:</p>
  
  <pre><code>br-ex     Link encap:Ethernet  HWaddr 1a:ba:3a:4a:d5:4f  
            inet addr:172.24.4.1  Bcast:0.0.0.0  Mask:255.255.255.0
            inet6 addr: fe80::18ba:3aff:fe4a:d54f/64 Scope:Link
            inet6 addr: 2001:db8::2/64 Scope:Global
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:15 errors:0 dropped:0 overruns:0 frame:0
            TX packets:79 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1 
            RX bytes:776 (776.0 B)  TX bytes:10328 (10.3 KB)
  </code></pre>
  
  <p>Not sure if the absense of this bridge (br-ex) could be the cause Horizon (from the Host) could not connect with the services in the Guest. Tried to configure statically in etc/network/interfaces as below to get the same above br-ex output.</p>
  
  <pre><code>auto br-ex
    iface br-ex inet static
    hwaddress 1a:ba:3a:4a:d5:4f
    address 172.24.4.1
    broadcast 0.0.0.0
    netmask 255.255.255.0
  
    iface br-ex inet6 static
    pre-up modprobe ipv6
    address 2001:db8::2
    netmask 64
  </code></pre>
  
  <p>The br-ex output I got after </p>
  
  <pre><code>svanan@svanan:~$ sudo systemctl restart networking
  svanan@svanan:~$ ifconfig
  
  br-ex     Link encap:Ethernet  HWaddr 7a:07:1d:1d:1a:40  
            inet addr:172.24.4.1  Bcast:0.0.0.0  Mask:255.255.255.0
            inet6 addr: fe80::18ba:3aff:fe4a:d54f/64 Scope:Link
            inet6 addr: 2001:db8::2/64 Scope:Global
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:126 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1 
            RX bytes:0 (0.0 B)  TX bytes:19334 (19.3 KB)
  </code></pre>
  
  <p>As can be seen above I am unable to configure HWaddr to 1a:ba:3a:4a:d5:4f instead 7a:07:1d:1d:1a:40 showing up.</p>
  
  <p><strong>Questions:</strong></p>
  
  <ol>
  <li>Could the absense of br-ex after the VM shutdown and switched on sequence could be the reason Horizon’s failure to retrieve the information from its services.</li>
  <li>How to statically set the HWaddr to 1a:ba:3a:4a:d5:4f for br-ex? The syntax hwaddress 1a:ba:3a:4a:d5:4f does not seem to work.</li>
  <li>If you faces similar problem, can you share how did you solve it.</li>
  <li><p>Is it true, one need to run ./unstack.sh and ./stack.sh ( ./rejoin-stack.sh has been depreciated in Newton) everytime a VM is restarted as mentioned in <a href="https://stackoverflow.com/questions/36268822/no-rejoin-stack-sh-script-in-my-setup">No rejoin-stack.sh script in my setup</a> .This approach takes very long time as shown below,</p>
  
  <p>=========================</p>
  
  <p>DevStack Component Timing</p>
  
  <p>=========================</p>
  
  <p>Total runtime         1777</p></li>
  </ol>
  
  <p>Hope you can assist to guide me in this. Thanks.</p>
  
  <p>Saravanan Kandasamy</p>