text:
"""
Simple server that just prints every message it receives.

    python simple_server.py localhost:8080
"""
import sys
from mido import sockets

if sys.argv[1:]:
    address = sys.argv[1]
else:
    address = 'localhost:9080'

try:
    (hostname, portno) = sockets.parse_address(address)
    print('Serving on {}'.format(address))
    with sockets.PortServer(hostname, portno) as server:
        for message in server:
            print(message)
except KeyboardInterrupt:
    pass


text:
This documentation was created to accompany the [Public Data Explorer (PDE)](http://www.google.com/publicdata) visualizations hosted at [measurementlab.net/visualization](http://measurementlab.net/visualization). Our aim is to expose the analytic methodology employed when computing these charts from the raw data. In addition, we hope to provide others interested in similar analysis and visualization with the tools they need to replicate and build on what we've done. If anything in this document is unclear, your feedback and questions at [measurementlab.net/contact](http://measurementlab.net/contact) would be greatly appreciated.

# Introduction #

All the charts published at [http://measurementlab.net/visualizations](http://measurementlab.net/visualizations) are computed from the [web100](http://web100.org/) logs collected by [NDT](http://measurementlab.net/measurement-lab-tools#ndt). The web page [https://developers.google.com/bigquery/docs/dataset-mlab](https://developers.google.com/bigquery/docs/dataset-mlab) explains what web100 logs are and how to access them.

The metrics visualized in the charts are computed as follows:
  1. Compute the metric values for every test, by running SQL queries against the [M-Lab tables in BigQuery](https://developers.google.com/bigquery/docs/dataset-mlab).
    * At this step, test results from incomplete tests are discarded.
  1. Geolocate the results of the query.
  1. Aggregate the results of the query by month, geography (city, region, country and world-wide) and ISP.
    * At each level of aggregation, locations with fewer than 200 distinct client IP addresses are not included in the charts.

This page describes these steps in detail.

# Short description of NDT #

This section is only meant to provide enough information about NDT to understand the rest of the document. We have created an illustration to clarify how NDT works.

![ndt_explanation.png](ndt_explanation.png)

For further information about NDT, see [here](http://measurementlab.net/measurement-lab-tools#ndt).

  * NDT is a client-server application that measures - among other things - the throughput of a single TCP connection, by transferring as much data as possible from the client to the server (client-to-server test) and from the server to client (server-to-client test) for at least 10 seconds, in each direction.
    * Results of NDT tests are indicated in BigQuery with `project = 0`
    * Results of client-to-server tests are indicated in BigQuery with `connection_spec.data_direction = 0`
    * Results of server-to-client tests are indicated in BigQuery with  `connection_spec.data_direction = 1`
  * To estimate the performance of a user connection, NDT attempts to _stress_ the connection, by creating congestion between the user's machine and an M-Lab server. An NDT test can end in 3 possible states:
    * The test ends during slow start and never reaches congestion. This can happen if the test was interrupted by the user or due to some errors. In this case, the test results underestimate the connection performance, because the test was not able to stress the connection. This condition is expressed in BigQuery with: `web100_log_entry.snap.CongSignals = 0`
    * The test ends after slow start, but during the first congestion episode. This can happen if the test was interrupted by the user or due to some errors. In this case, the peak values in the test results are valid data points, while average values are not. This condition is expressed in BigQuery with:`web100_log_entry.snap.CongSignals = 1`
    * The test ends after the first congestion episode. In this case, the both peak values and averages in the test results are valid data points. This condition is expressed in BigQuery with: `web100_log_entry.snap.CongSignals > 1`
  * During every test, the values of the web100 variables are incremented when TCP-related events occur. NDT polls the values of all the web100 variables every 5ms and writes them into a log file. Every test has a separate log file. The last entry of this log accounts for the whole test and contains the snapshot of web100 variable values that is most commonly used for analysis of NDT tests.
    * Test results extracted from the last line of every web100 log are indicated in BigQuery with `web100_log_entry.is_last_entry = True`
  * The web100 logs used for the analysis are collected only on the server side, for both server-to-client and client-to-server tests. As a consequence, the web100 variables used to study client-to-server tests are not always the same variables used to study server-to-client tests. For example,
    * The duration of a server-to-client test is `web100_log_entry.snap.SndLimTimeRwin + web100_log_entry.snap.SndLimTimeCwnd + web100_log_entry.snap.SndLimTimeSnd`
      * This accounts for the time interval between when the TCP connection was created and the last TCP event recorded.
    * The duration of a client-to-server test is `web100_log_entry.snap.Duration`
      * This accounts for the time interval between when the TCP connection was created and the last time NDT polled the web100 variable values. This may include some idle time at the end of the connection. However, it's not possible to get a more accurate estimate of the duration of client-to-server tests, because `web100_log_entry.snap.SndLimTimeRwin, web100_log_entry.snap.SndLimTimeCwnd, web100_log_entry.snap.SndLimTimeSnd` don't record the duration of client-to-server tests.
    * The data transferred during of a client-to-server test is `web100_log_entry.snap.HCThruOctetsReceived` We use HCThruOctetsReceived rather than ThruOctetsReceived because HCThruOctetsReceived is encoded with 64 bits, as opposed to ThruOctetsReceived which is only encoded with 32 bits and is prone to overflow.
    * The data transferred during of a server-to-client test is `web100_log_entry.snap.HCThruOctetsAcked` We use HCThruOctetsAcked rather than ThruOctetsAcked because HCThruOctetsAcked is  encoded with 64 bits, as opposed to ThruOctetsAcked which is only encoded with 32 bits and is prone to overflow.
  * Most of the web100 variables are incremented when data are sent and not when data are received. As a consequence, it's possible to compute some metrics only for the server-to-client tests and not for the client-to-server tests.

## Incomplete tests ##

In the following, a test is defined **incomplete** if it was interrupted by the user or due to some errors
  * Before the measurement ended, or
  * Before the test completed logging the measurement data.

Complete tests are identified by the following criteria:
  1. The test lasted longer than **9 seconds**.
    * Allowed some flexibility, to account for tests that lasted _almost_ 10 seconds.
    * This condition is expressed in BigQuery
      * For client-to-server tests: `web100_log_entry.snap.Duration >= 9000000`
      * For server-to-client tests: `web100_log_entry.snap.SndLimTimeRwin + web100_log_entry.snap.SndLimTimeCwnd + web100_log_entry.snap.SndLimTimeSnd >= 9000000`
    * We also exclude results from tests that lasted much longer than expected (e.g., 10 min), because this is likely a symptom of problems during the test run.
  1. The test exchanged at least **8192 bytes**.
    * This excludes results from users with dial-up connections and instances where MTU negotiation fails.
    * This condition is expressed in BigQuery
      * For client-to-server tests: `web100_log_entry.snap.HCThruOctetsReceived >= 8192`
      * For server-to-client tests: `web100_log_entry.snap.HCThruOctetsAcked >= 8192`
  1. The test concluded the 3-way-handshake and the connection was **established** (and possibly **closed**).
    * See more information about TCP states and how they are recorded in the web100 logs, see http://www.web100.org/download/kernel/tcp-kis.txt
    * This condition is expressed in BigQuery with: `web100_log_entry.snap.State == 1 || (web100_log_entry.snap.State >= 5  && web100_log_entry.snap.State <= 11)`

# Compute test-level metrics #

For each metric, this section includes the BigQuery queries used to compute the metrics.
All the queries include conditions to filter out incomplete test results.

## Download throughput ##

The download throughput is computed for every server-to-client test as the ratio of the data transmitted during the test and the duration of the test. We multiply by 8 to convert from octets (bytes) to bits:

```
8 *
(web100_log_entry.snap.HCThruOctetsAcked /
 (web100_log_entry.snap.SndLimTimeRwin +
 web100_log_entry.snap.SndLimTimeCwnd +
 web100_log_entry.snap.SndLimTimeSnd))
```

Results of tests that ended during **slow start** are excluded.

The complete BigQuery query is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 8 * (web100_log_entry.snap.HCThruOctetsAcked /
      (web100_log_entry.snap.SndLimTimeRwin +
       web100_log_entry.snap.SndLimTimeCwnd +
       web100_log_entry.snap.SndLimTimeSnd)) AS download_Mbps
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.CongSignals)
 AND web100_log_entry.snap.CongSignals > 0
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

## Upload throughput ##

The upload throughput is computed for every client-to-server test as the ratio of the data transmitted during the test and the duration of the test. We multiply by 8 to convert from octets (bytes) to bits: `8 * (web100_log_entry.snap.HCThruOctetsReceived/web100_log_entry.snap.Duration)`

It is not possible to exclude results of tests that ended during slow start, because the web100 variable `web100_log_entry.snap.CongSignals` is not updated during client-to-server tests.

The complete BigQuery query is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 8 * (web100_log_entry.snap.HCThruOctetsReceived/web100_log_entry.snap.Duration) AS upload_Mbps
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsReceived)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.Duration)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 0
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsReceived >= 8192
 AND web100_log_entry.snap.Duration >= 9000000
 AND web100_log_entry.snap.Duration < 3600000000
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

## Round Trip Time (RTT) ##

Server-to-client RTT is affected by TCP congestion. As a consequence, there are (at least) 2 ways to estimate the RTT using the web100 data. These 2 ways provide different, non-equivalent information about the user connection.
  1. Server-client **(time) distance**
    * Estimated using the **minimum RTT** measured during the test, which most likely happened before the test reached congestion.
    * This value is reported by the web100 variable `web100_log_entry.snap.MinRTT`
    * However, using this variable has the drawback that it might underestimate the connection RTT, because it might be measured in the SYC ACK exchange or some other tiny transaction which, for low speed links, does not represent the typical RTT for the full data segment.
    * Note that using `PreCongSumRTT/PreCongCountRTT` does not provide a more accurate estimate, because both `PreCongSumRTT` and `PreCongCountRTT` are recorded right before the first congestion signal, which, in the worst case, occurs when the receiver queue is already full, which affects the RTT.
  1. Server-client **latency during data transfers** (with congestion)
    * Estimated using the **average RTT**, uniformly averaged over an entire test.
    * This value can be computed as `web100_log_entry.snap.SumRTT/web100_log_entry.snap.CountRTT`
    * In this case, it makes sense to exclude results of tests with **10 or fewer round trip time samples**, because there are not enough samples to accurately estimate the RTT. This condition is expressed in BigQuery with:`web100_log_entry.snap.CountRTT > 10`

Given that the NDT server updates the web100 variables `web100_log_entry.snap.MinRTT` and `web100_log_entry.snap.CountRTT` only when it receives an acknowledgement and given that, during client-to-server tests the NDT server receives an ack only during the 3-way-handshake, RTT values are computed only for server-to-client tests.

The complete BigQuery query is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 web100_log_entry.snap.MinRTT AS min_rtt
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.MinRTT)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.CountRTT)
 AND web100_log_entry.snap.CountRTT > 10
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

## Number of tests ##

The number of tests counts all the server-to-client tests whose results meet the same criteria used to compute the download throughput.
As a consequence, the number of tests is computed as the number of entries in the result of the BigQuery query in the [Download throughput section](#Download_throughput.md).

## Percentage of tests that reached congestion ##

The percentage of tests that reached congestion is computed as the ratio between the number of tests that reached congestion and the number of all the tests, where
  * The number of tests that reached congestion is computed as the number of entries in the result of the BigQuery query in the [Download throughput section](#Download_throughput.md), with the additional condition `web100_log_entry.snap.CongSignals > 1`
    * Given that the NDT server updates the web100 variable `web100_log_entry.snap.CongSignals` only when the server receives a congestion signal, the number of tests that reached congestion is reported for only server-to-client tests.
  * The number of all the tests is computed as described in the [Number of tests section](#Number_of_tests.md).

## Packet retransmission ##

The NDT keeps track of the number of packets retransmitted during a test, in the web100 variable `web100_log_entry.snap.SegsRetrans`.

Packet retransmission is computed as the ratio between the re-transmitted packets and all the transmitted packets.
`web100_log_entry.snap.SegsRetrans/web100_log_entry.snap.DataSegsOut`

Given that the NDT server updates the web100 variables `web100_log_entry.snap.SegsRetrans` and `web100_log_entry.snap.DataSegsOut` only when sending data, packet retransmission is only estimated for server-to-client tests.

It is possible to also measure the byte retransmission, as `web100_log_entry.snap.OctetsRetrans/web100_log_entry.snap.DataOctetsOut`

The complete BigQuery query is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 (web100_log_entry.snap.SegsRetrans / web100_log_entry.snap.DataSegsOut) AS packet_retransmission_rate
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SegsRetrans)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.DataSegsOut)
 AND web100_log_entry.snap.DataSegsOut > 0
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

## Network-limited ratio and client-limited time ratio ##

An NDT test can be in 3 different states. Each state represents different conditions that limit the data sent by the server to the client.
  * **network-limited**, when the network is congested.
    * The web00 variable `web100_log_entry.snap.SndLimTimeCwnd` reports the time spent in this state.
  * **receiver-limited**, when the receiver (client) limits the data that can be received.
    * The web00 variable `web100_log_entry.snap.SndLimTimeRwin` reports the time spent in this state.
  * **server-limited**, when the server limits the data that can be sent.
    * The web00 variable `web100_log_entry.snap.SndLimTimeSnd` reports the time spent in this state.
    * This state can happen because
      * The uplink is congested. Note however that M-Lab servers are specially (over)provisioned to avoid this case.
      * The server cannot send as much data as the network and the receiver would allow, in specific phases of the tests. This usually happens during slow start, especially with fast networks and high values of initial window.
      * The server cannot send as much data as the network and the receiver would allow, during the whole test. This can happen for specific TCP configurations (e.g., if GSO is on). However, this kind of configuration is disabled by default on M-Lab servers.

The complete BigQuery query to compute network-limited time is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip,
 web100_log_entry.connection_spec.local_ip,
 web100_log_entry.snap.SndLimTimeCwnd /
   (web100_log_entry.snap.SndLimTimeRwin +
    web100_log_entry.snap.SndLimTimeCwnd +
    web100_log_entry.snap.SndLimTimeSnd) AS network_limited_time
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
AND (web100_log_entry.snap.State == 1
     OR (web100_log_entry.snap.State >= 5
         AND web100_log_entry.snap.State <= 11))
```

The complete BigQuery query to compute receiver-limited time is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 web100_log_entry.snap.SndLimTimeRwin /
   (web100_log_entry.snap.SndLimTimeRwin +
    web100_log_entry.snap.SndLimTimeCwnd +
    web100_log_entry.snap.SndLimTimeSnd) AS receiver_limited_time
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

## Receiver window scale ##

The receiver window scale is the value negotiated at the beginning of a TCP connection to scale the receiver window size. The receive window size is the maximum amount of received data that can be buffered at one time on the receiving side of a TCP connection.
The value of receiver window scale depends on the type and the version of the client's operating system. As a consequence, the distribution of receiver window scale values shows the distribution of operating systems among NDT users.

The receiver window scale of a test is the value of the web100 variable `web100_log_entry.snap.WinScaleRcvd`.
As described in the [web100 variable definition](http://www.web100.org/download/kernel/tcp-kis.txt), the valid values of `web100_log_entry.snap.WinScaleRcvd` are (-1 .. 14), where -1 means that the receiver did not request any value.

The complete BigQuery query is:
```sql
SELECT
 web100_log_entry.connection_spec.remote_ip AS remote_ip,
 web100_log_entry.connection_spec.local_ip AS local_ip,
 web100_log_entry.snap.WinScaleRcvd
FROM
 [plx.google:m_lab.YYYY_MM.all]
WHERE
 IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.remote_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.connection_spec.local_ip)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.HCThruOctetsAcked)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeRwin)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeCwnd)
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.SndLimTimeSnd)
 AND project = 0
 AND IS_EXPLICITLY_DEFINED(connection_spec.data_direction)
 AND connection_spec.data_direction = 1
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.is_last_entry)
 AND web100_log_entry.is_last_entry = True
 AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) >= 9000000
 AND (web100_log_entry.snap.SndLimTimeRwin +
      web100_log_entry.snap.SndLimTimeCwnd +
      web100_log_entry.snap.SndLimTimeSnd) < 3600000000
 AND IS_EXPLICITLY_DEFINED(web100_log_entry.snap.WinScaleRcvd)
 AND web100_log_entry.snap.WinScaleRcvd >= -1
 AND web100_log_entry.snap.WinScaleRcvd <= 14
 AND (web100_log_entry.snap.State == 1
      OR (web100_log_entry.snap.State >= 5
          AND web100_log_entry.snap.State <= 11))
```

# Aggregate query results #

Query results computed as described in the previous section are aggregated using various functions.
This section describes the aggregation functions and how they are applied to the query results to compute the data visualized in the Public Data Explorer charts.

## Dimensions ##

All the metrics are aggregated along the following dimensions:
  * Geography (country, region, city)
  * M-Lab site (see the [list of all the M-Lab sites](http://measurementlab.net/mlab_sites))
  * ISP
    * Every test is assigned to the AS (Autonomous System) that originates the client IP address.
    * All the ASes are aggregated into ISPs, according to the AS names published at [bgp.potaroo.net/cidr/autnums.html bgp.potaroo.net/cidr/autnums.html]. This [file](https://storage.cloud.google.com/m-lab/isp_to_asns_map.txt) lists all the ISPs visualized in the charts and, for each ISP, it lists all the ASes that are assigned to the ISP.

## Aggregation functions ##

### Median ###

Values of download throughput, upload throughput, and RTT are aggregated using the median function.

Note that, at every level of aggregation, the median is computed across the non aggregated values. For example, the median download throughput of country X is computed as the median of all the download throughput values geolocated within the country X (and not as the median of the download throughput medians of all the regions within X).

### Average ###

Values of receiver window scale are aggregated using the average function, weighted by the number of tests.

### Probability ###

Values of packet retrasmission, network-limited time ratio and client-limited time ratio at every level of aggregation are computed as the ration between two sums, computed across all the tests in the same aggregation bin.
In particular:

* Packet retransmission = `SUM(web100_log_entry.snap.SegsRetrans) /
SUM(web100_log_entry.snap.DataSegsOut)`
* Network-limited time ratio = `SUM(web100_log_entry.snap.SndLimTimeCwnd) /
SUM(web100_log_entry.snap.SndLimTimeRwin +
web100_log_entry.snap.SndLimTimeCwnd +
web100_log_entry.snap.SndLimTimeSnd)`
* Client-limited time ratio = `SUM(web100_log_entry.snap.SndLimTimeRwin) /
SUM(web100_log_entry.snap.SndLimTimeRwin +
web100_log_entry.snap.SndLimTimeCwnd +
web100_log_entry.snap.SndLimTimeSnd)`

redpajama_set_name: RedPajamaGithub


text:
use chrono::Local;
use getopts::{Fail, HasArg, Occur, Options};
use hyper_tls::HttpsConnector;
use std::default::Default;
use std::env;
use std::thread::sleep;
use std::time::Duration;
use yup_oauth2::{self as oauth2, GetToken};

fn usage(program: &str, opts: &Options, err: Option<Fail>) -> ! {
    if err.is_some() {
        println!("{}", err.unwrap());
        std::process::exit(1);
    }
    println!("{}", opts.short_usage(program) + " SCOPE [SCOPE ...]");
    println!(
        "{}",
        opts.usage(
            "A program to authenticate against oauthv2 services.\n\
             See https://developers.google.com/youtube/registering_an_application\n\
             and https://developers.google.com/youtube/v3/guides/authentication#devices"
        )
    );

    std::process::exit(0);
}

fn main() {
    let args: Vec<String> = env::args().collect();
    let prog = args[0].clone();

    let mut opts = Options::new();
    opts.opt(
        "c",
        "id",
        "oauthv2 ID of your application",
        "CLIENT_ID",
        HasArg::Yes,
        Occur::Req,
    )
    .opt(
        "s",
        "secret",
        "oauthv2 secret of your application",
        "CLIENT_SECRET",
        HasArg::Yes,
        Occur::Req,
    );

    let m = match opts.parse(&args[1..]) {
        Ok(m) => m,
        Err(e) => {
            usage(&prog, &opts, Some(e));
        }
    };

    if m.free.len() == 0 {
        let msg = Fail::ArgumentMissing(
            "you must provide one or more authorization scopes as free options".to_string(),
        );
        usage(&prog, &opts, Some(msg));
    }

    let secret = oauth2::ApplicationSecret {
        client_id: m.opt_str("c").unwrap(),
        client_secret: m.opt_str("s").unwrap(),
        token_uri: Default::default(),
        auth_uri: Default::default(),
        redirect_uris: Default::default(),
        ..Default::default()
    };

    println!("THIS PROGRAM PRINTS ALL COMMUNICATION TO STDERR !!!");

    struct StdoutHandler;
    impl oauth2::AuthenticatorDelegate for StdoutHandler {
        fn present_user_code(&mut self, pi: &oauth2::PollInformation) {
            println!(
                "Please enter '{}' at {} and authenticate the application for the\n\
                      given scopes. This is not a test !\n\
                      You have time until {} to do that.
                      Do not terminate the program until you deny or grant access !",
                pi.user_code,
                pi.verification_url,
                pi.expires_at.with_timezone(&Local)
            );
            let delay = Duration::from_secs(5);
            println!("Browser opens automatically in {:?} seconds", delay);
            sleep(delay);
            open::that(&pi.verification_url).ok();
            println!("DONE - waiting for authorization ...");
        }
    }

    let https = HttpsConnector::new(4).unwrap();
    let client = hyper::Client::builder().build(https);

    match oauth2::Authenticator::new(&secret, StdoutHandler, client, oauth2::NullStorage, None)
        .token(&m.free)
    {
        Ok(t) => {
            println!("Authentication granted !");
            println!("You should store the following information for use, or revoke it.");
            println!("All dates are given in UTC.");
            println!("{:?}", t);
        }
        Err(err) => {
            println!("Access token wasn't obtained: {}", err);
            std::process::exit(10);
        }
    };
}


text:
<?php

/*
 * This file is part of TravelBuilder v0.4 project
 */

namespace Compassites\ItineraryValidationHelper;

use Compassites\DateHelper\DateHelper;

/**
 * It helps to validate itinerary data
 *
 * @package ItineraryValidationHelper
 * @author Jeevan N <jeeeevz@gmail.com>
 * @version 1.0
 */
class ItineraryValidationHelper
{

    protected $dateHelper;
    public $totalCityStayDuration;
    public $itineraryStayDuration;
    public $isValid;
    public $hasErrors = false;
    private $validationType;

    public function __construct(DateHelper $dateHelper)
    {
        $this->dateHelper = $dateHelper;
    }

    public function validateItinerarySpanAgainstStaySpanInEachCity($sessionData)
    {
        $this->validationType = 1;
        $itineraryStayDuration = $sessionData['itinerary']['nights'];
        $totalCityStayDuration = 0;
        foreach ($sessionData['city'] as $city) {
            $totalCityStayDuration+=$city['nightCount'];
        }
        $this->totalCityStayDuration = (int) $totalCityStayDuration;
        $this->itineraryStayDuration = (int) $itineraryStayDuration;
        return $this->isValid = $this->totalCityStayDuration <= $this->itineraryStayDuration;
    }

    function getErrorMsg()
    {
        if ($this->validationType == 1) {
            return "Your stay should not exceed {$this->itineraryStayDuration} days, but is {$this->totalCityStayDuration} days";
        }
        if($this->validationType ==2){
             return $this->errorArray;
        }
    }

    public function validateItineraryHotelPriceForZeroValues($sessionData)
    {
        $this->validationType =2;
        $errorMsg = "Price for the hotel %hotel_name% is zero";
        $errorArray['warning'] = [];
        foreach ($sessionData['city'] as $index => $city) {
            foreach (array('optionone', 'optiontwo') as $option) {
                $hotel_tsid = array_get($city, "{$option}.hotel.hotel_tsid", null);
                if ($hotel_tsid && (int) $hotel_tsid > 0) {
                    $selectedHotelPrice = array_get($city, "{$option}.hotel.selectedHotelPrice", 0);
                    $selectedHotelName = array_get($city, "{$option}.hotel.hotel_name", '');
                    if (!((int)$selectedHotelPrice > 0)) {
                       $this->hasErrors = true;
                       $errorMsg = str_replace("%hotel_name%", $selectedHotelName, $errorMsg);
                       $errorArray['warning'][$index]['hotel'][$hotel_tsid]=[$errorMsg];
                    }
                }
            }
        }
        $this->errorArray = $errorArray;
        return array_merge($sessionData,$errorArray);
    }

}


text:
import os
import logging
import time

from lib.cuckoo.common.abstracts import Processing
from lib.cuckoo.common.config import Config
from lib.cuckoo.common.constants import CUCKOO_ROOT

log = logging.getLogger(__name__)

try:
    import volatility.conf as conf
    import volatility.registry as registry
    import volatility.commands as commands
    import volatility.utils as utils
    import volatility.plugins.malware.devicetree as devicetree
    import volatility.plugins.malware.apihooks as apihooks
    import volatility.plugins.getsids as sidm
    import volatility.plugins.privileges as privm
    import volatility.plugins.taskmods as taskmods
    import volatility.win32.tasks as tasks
    import volatility.obj as obj
    import volatility.exceptions as exc
    import volatility.plugins.filescan as filescan
    import volatility.protos as protos

    HAVE_VOLATILITY = True

    # Inherit Cuckoo debugging level for Volatility commands.
    rootlogger = logging.getLogger()
    logging.getLogger("volatility.obj").setLevel(rootlogger.level)
    logging.getLogger("volatility.utils").setLevel(rootlogger.level)
except ImportError as e:
    if e.message == "No module named Crypto.Hash":
        log.error(
            "The PyCrypto package is missing (install with "
            "`pip install pycrypto`)"
        )

    HAVE_VOLATILITY = False

class VolatilityAPI(object):
    """ Volatility API interface."""

    def __init__(self, memdump, osprofile=None):
        """@param memdump: the memdump file path
        @param osprofile: the profile (OS type)
        """
        registry.PluginImporter()
        self.memdump = memdump
        self.osprofile = osprofile
        self.config = None
        self.addr_space = None
        self.init_config()

    def get_dtb(self):
        """Use psscan to get system dtb and apply it."""
        ps = filescan.PSScan(self.config)

        for ep in ps.calculate():
            if str(ep.ImageFileName) == "System":
                self.config.update("dtb", ep.Pcb.DirectoryTableBase)
                return True

        return False

    def init_config(self):
        """Creates a volatility configuration."""
        if self.config is not None and self.addr_space is not None:
            return self.config

        self.config = conf.ConfObject()
        self.config.optparser.set_conflict_handler("resolve")
        registry.register_global_options(self.config, commands.Command)
        base_conf = {
            "profile": "WinXPSP2x86",
            "use_old_as": None,
            "kdbg": None,
            "help": False,
            "kpcr": None,
            "tz": None,
            "pid": None,
            "output_file": None,
            "physical_offset": None,
            "conf_file": None,
            "dtb": None,
            "output": None,
            "info": None,
            "location": "file://" + self.memdump,
            "plugins": None,
            "debug": None,
            "cache_dtb": True,
            "filename": None,
            "cache_directory": None,
            "verbose": None,
            "write": False
        }

        if self.osprofile:
            base_conf["profile"] = self.osprofile

        for key, value in base_conf.items():
            self.config.update(key, value)

        # Deal with Volatility support for KVM/qemu memory dump.
        # See: #464.
        try:
            self.addr_space = utils.load_as(self.config)
        except exc.AddrSpaceError:
            if self.get_dtb():
                self.addr_space = utils.load_as(self.config)
            else:
                raise

        self.plugins = \
            registry.get_plugin_classes(commands.Command, lower=True)
        return self.config

    def pslist(self):
        """Volatility pslist plugin.
        @see volatility/plugins/taskmods.py
        """
        results = []

        command = taskmods.PSList(self.config)
        for process in command.calculate():
            new = {
                "process_name": str(process.ImageFileName),
                "process_id": int(process.UniqueProcessId),
                "parent_id": int(process.InheritedFromUniqueProcessId),
                "num_threads": str(process.ActiveThreads),
                "num_handles": str(process.ObjectTable.HandleCount),
                "session_id": str(process.SessionId),
                "create_time": str(process.CreateTime or ""),
                "exit_time": str(process.ExitTime or ""),
            }

            results.append(new)

        return dict(config={}, data=results)

    def psxview(self):
        """Volatility psxview plugin.
        @see volatility/plugins/malware/psxview.py
        """
        results = []

        command = self.plugins["psxview"](self.config)
        for offset, process, ps_sources in command.calculate():
            new = {
                "process_name": str(process.ImageFileName),
                "process_id": int(process.UniqueProcessId),
                "pslist": str(offset in ps_sources["pslist"]),
                "psscan": str(offset in ps_sources["psscan"]),
                "thrdproc": str(offset in ps_sources["thrdproc"]),
                "pspcid": str(offset in ps_sources["pspcid"]),
                "csrss": str(offset in ps_sources["csrss"]),
                "session": str(offset in ps_sources["session"]),
                "deskthrd": str(offset in ps_sources["deskthrd"]),
            }

            results.append(new)

        return dict(config={}, data=results)

    def callbacks(self):
        """Volatility callbacks plugin.
        @see volatility/plugins/malware/callbacks.py
        """
        results = []

        command = self.plugins["callbacks"](self.config)
        for (sym, cb, detail), mods, mod_addrs in command.calculate():
            module = tasks.find_module(mods, mod_addrs, self.addr_space.address_mask(cb))

            if module:
                module_name = module.BaseDllName or module.FullDllName
            else:
                module_name = "UNKNOWN"

            new = {
                "type": str(sym),
                "callback": hex(int(cb)),
                "module": str(module_name),
                "details": str(detail or "-"),
            }

            results.append(new)

        return dict(config={}, data=results)

    def idt(self):
        """Volatility idt plugin.
        @see volatility/plugins/malware/idt.py
        """
        results = []

        command = self.plugins["idt"](self.config)
        for n, entry, addr, module in command.calculate():
            if module:
                module_name = str(module.BaseDllName or "")
                sect_name = command.get_section_name(module, addr)
            else:
                module_name = "UNKNOWN"
                sect_name = ''

            # The parent is IDT. The grand-parent is _KPCR.
            cpu_number = entry.obj_parent.obj_parent.ProcessorBlock.Number
            new = {
                "cpu_number": int(cpu_number),
                "index": int(n),
                "selector": hex(int(entry.Selector)),
                "address": hex(int(addr)),
                "module": module_name,
                "section": sect_name,
            }
            results.append(new)

        return dict(config={}, data=results)

    def gdt(self):
        """Volatility gdt plugin.
        @see volatility/plugins/malware/idt.py
        """
        results = []

        command = self.plugins["gdt"](self.config)
        # Comment: this code is pretty much ripped from render_text in volatility.
        for n, entry in command.calculate():
            selector = n * 8

            # Is the entry present? This applies to all types of GDT entries
            if entry.Present:
                present = "P"
            else:
                present = "Np"

            # The base, limit, and granularity is calculated differently
            # for 32bit call gates than they are for all other types.
            if entry.Type == "CallGate32":
                base = entry.CallGate
                limit = 0
                granularity = "-"
            else:
                base = entry.Base
                limit = entry.Limit
                if entry.Granularity:
                    granularity = "Pg"
                else:
                    granularity = "By"

            # The parent is GDT. The grand-parent is _KPCR.
            cpu_number = entry.obj_parent.obj_parent.ProcessorBlock.Number

            new = {
                "cpu_number": int(cpu_number),
                "selector": hex(selector),
                "base": hex(int(base)),
                "limit": hex(int(limit)),
                "type": str(entry.Type),
                "dpl": str(entry.Dpl),
                "granularity": granularity,
                "present": present,
            }
            results.append(new)

        return dict(config={}, data=results)

    def ssdt(self):
        """Volatility ssdt plugin.
        @see volatility/plugins/ssdt.py
        """
        results = []

        command = self.plugins["ssdt"](self.config)

        # Comment: this code is pretty much ripped from render_text in volatility.
        addr_space = self.addr_space
        syscalls = addr_space.profile.syscalls
        bits32 = addr_space.profile.metadata.get("memory_model", "32bit") == "32bit"

        for idx, table, n, vm, mods, mod_addrs in command.calculate():
            for i in range(n):
                if bits32:
                    # These are absolute function addresses in kernel memory.
                    syscall_addr = obj.Object("address", table + (i * 4), vm).v()
                else:
                    # These must be signed long for x64 because they are RVAs relative
                    # to the base of the table and can be negative.
                    offset = obj.Object("long", table + (i * 4), vm).v()
                    # The offset is the top 20 bits of the 32 bit number.
                    syscall_addr = table + (offset >> 4)

                try:
                    syscall_name = syscalls[idx][i]
                except IndexError:
                    syscall_name = "UNKNOWN"

                syscall_mod = tasks.find_module(mods, mod_addrs, addr_space.address_mask(syscall_addr))
                if syscall_mod:
                    syscall_modname = "{0}".format(syscall_mod.BaseDllName)
                else:
                    syscall_modname = "UNKNOWN"

                new = {
                    "index": int(idx),
                    "table": "0x%x" % int(table),
                    "entry": "{0:#06x}".format(idx * 0x1000 + i),
                    "syscall_name": syscall_name,
                    "syscall_addr": "0x%x" % int(syscall_addr),
                    "syscall_modname": syscall_modname,
                }

                if bits32 and syscall_mod is not None:
                    ret = apihooks.ApiHooks.check_inline(
                        va=syscall_addr, addr_space=vm,
                        mem_start=syscall_mod.DllBase,
                        mem_end=syscall_mod.DllBase + syscall_mod.SizeOfImage)

                    # Could not analyze the memory.
                    if ret is not None:
                        hooked, data, dest_addr = ret
                        if hooked:
                            # We found a hook, try to resolve the hooker.
                            # No mask required because we currently only work
                            # on x86 anyway.
                            hook_mod = tasks.find_module(mods, mod_addrs,
                                                         dest_addr)
                            if hook_mod:
                                hook_name = "{0}".format(hook_mod.BaseDllName)
                            else:
                                hook_name = "UNKNOWN"

                            # Report it now.
                            new.update({
                                "hook_dest_addr": "{0:#x}".format(dest_addr),
                                "hook_name": hook_name,
                            })

                results.append(new)

        return dict(config={}, data=results)

    def timers(self):
        """Volatility timers plugin.
        @see volatility/plugins/malware/timers.py
        """
        results = []

        command = self.plugins["timers"](self.config)
        for timer, module in command.calculate():
            if timer.Header.SignalState.v():
                signaled = "Yes"
            else:
                signaled = "-"

            if module:
                module_name = str(module.BaseDllName or "")
            else:
                module_name = "UNKNOWN"

            due_time = "{0:#010x}:{1:#010x}".format(timer.DueTime.HighPart, timer.DueTime.LowPart)

            new = {
                "offset": hex(timer.obj_offset),
                "due_time": due_time,
                "period": int(timer.Period),
                "signaled": signaled,
                "routine": hex(int(timer.Dpc.DeferredRoutine)),
                "module": module_name,
            }
            results.append(new)

        return dict(config={}, data=results)

    def messagehooks(self):
        """Volatility messagehooks plugin.
        @see volatility/plugins/malware/messagehooks.py
        """
        results = []

        command = self.plugins["messagehooks"](self.config)
        for winsta, atom_tables in command.calculate():
            for desk in winsta.desktops():
                for name, hook in desk.hooks():
                    module = command.translate_hmod(winsta, atom_tables, hook.ihmod)
                    new = {
                        "offset": hex(int(hook.obj_offset)),
                        "session": int(winsta.dwSessionId),
                        "desktop": "{0}\\{1}".format(winsta.Name, desk.Name),
                        "thread": "<any>",
                        "filter": str(name),
                        "flags": str(hook.flags),
                        "function": hex(int(hook.offPfn)),
                        "module": str(module),
                    }
                    results.append(new)

                for thrd in desk.threads():
                    info = "{0} ({1} {2})".format(
                        thrd.pEThread.Cid.UniqueThread,
                        thrd.ppi.Process.ImageFileName,
                        thrd.ppi.Process.UniqueProcessId)

                    for name, hook in thrd.hooks():
                        module = command.translate_hmod(winsta, atom_tables, hook.ihmod)

                        new = {
                            "offset": hex(int(hook.obj_offset)),
                            "session": int(winsta.dwSessionId),
                            "desktop": "{0}\\{1}".format(winsta.Name, desk.Name),
                            "thread": str(info),
                            "filter": str(name),
                            "flags": str(hook.flags),
                            "function": hex(int(hook.offPfn)),
                            "module": str(module),
                        }
                        results.append(new)

        return dict(config={}, data=results)

    def getsids(self):
        """Volatility getsids plugin.
        @see volatility/plugins/malware/getsids.py
        """
        results = []

        command = self.plugins["getsids"](self.config)
        for task in command.calculate():
            token = task.get_token()

            if not token:
                continue

            for sid_string in token.get_sids():
                if sid_string in sidm.well_known_sids:
                    sid_name = " {0}".format(sidm.well_known_sids[sid_string])
                else:
                    sid_name_re = sidm.find_sid_re(sid_string, sidm.well_known_sid_re)
                    if sid_name_re:
                        sid_name = " {0}".format(sid_name_re)
                    else:
                        sid_name = ""

                new = {
                    "filename": str(task.ImageFileName),
                    "process_id": int(task.UniqueProcessId),
                    "sid_string": str(sid_string),
                    "sid_name": str(sid_name),
                }
                results.append(new)

        return dict(config={}, data=results)

    def privs(self):
        """Volatility privs plugin.
        @see volatility/plugins/malware/privs.py
        """
        results = []

        command = self.plugins["privs"](self.config)

        for task in command.calculate():
            for value, present, enabled, default in task.get_token().privileges():
                try:
                    name, desc = privm.PRIVILEGE_INFO[int(value)]
                except KeyError:
                    continue

                attributes = []
                if present:
                    attributes.append("Present")
                if enabled:
                    attributes.append("Enabled")
                if default:
                    attributes.append("Default")

                new = {
                    "process_id": int(task.UniqueProcessId),
                    "filename": str(task.ImageFileName),
                    "value": int(value),
                    "privilege": str(name),
                    "attributes": ",".join(attributes),
                    "description": str(desc),
                }
                results.append(new)

        return dict(config={}, data=results)

    def malfind(self, dump_dir=None):
        """Volatility malfind plugin.
        @param dump_dir: optional directory for dumps
        @see volatility/plugins/malware/malfind.py
        """
        results = []

        command = self.plugins["malfind"](self.config)
        for task in command.calculate():
            for vad, address_space in task.get_vads(vad_filter=task._injection_filter):
                if command._is_vad_empty(vad, address_space):
                    continue

                new = {
                    "process_name": str(task.ImageFileName),
                    "process_id": int(task.UniqueProcessId),
                    "vad_start": "{0:#x}".format(vad.Start),
                    "vad_tag": str(vad.Tag),
                }
                results.append(new)

                if dump_dir:
                    filename = os.path.join(dump_dir, "process.{0:#x}.{1:#x}.dmp".format(task.obj_offset, vad.Start))
                    command.dump_vad(filename, vad, address_space)

        return dict(config={}, data=results)

    def yarascan(self):
        """Volatility yarascan plugin.
        @see volatility/plugins/malware/yarascan.py
        """
        results = []

        ypath = os.path.join(CUCKOO_ROOT, "data", "yara", "index_memory.yar")
        if not os.path.exists(ypath):
            return dict(config={}, data=[])

        self.config.update("YARA_FILE", ypath)

        command = self.plugins["yarascan"](self.config)
        for o, addr, hit, content in command.calculate():
            # Comment: this code is pretty much ripped from render_text in volatility.
            # Find out if the hit is from user or kernel mode
            if o is None:
                owner = "Unknown Kernel Memory"
            elif o.obj_name == "_EPROCESS":
                owner = "Process {0} Pid {1}".format(o.ImageFileName, o.UniqueProcessId)
            else:
                owner = "{0}".format(o.BaseDllName)

            hexdump = "".join(
                "{0:#010x}  {1:<48}  {2}\n".format(addr + o, h, ''.join(c))
                for o, h, c in utils.Hexdump(content[0:64]))

            new = {
                "rule": hit.rule,
                "owner": owner,
                "hexdump": hexdump,
            }
            results.append(new)

        return dict(config={}, data=results)

    def apihooks(self):
        """Volatility apihooks plugin.
        @see volatility/plugins/malware/apihooks.py
        """
        results = []

        command = self.plugins["apihooks"](self.config)
        for process, module, hook in command.calculate():
            proc_name = str(process.ImageFileName) if process else ''
            if command.whitelist(hook.hook_mode | hook.hook_type,
                                 proc_name, hook.VictimModule,
                                 hook.HookModule, hook.Function):
                continue

            new = {
                "hook_mode": str(hook.Mode),
                "hook_type": str(hook.Type),
                "victim_module": str(module.BaseDllName or ""),
                "victim_function": str(hook.Detail),
                "hook_address": "{0:#x}".format(hook.hook_address),
                "hooking_module": str(hook.HookModule)
            }

            if process:
                new["process_id"] = int(process.UniqueProcessId)
                new["process_name"] = str(process.ImageFileName)

            results.append(new)

        return dict(config={}, data=results)

    def dlllist(self):
        """Volatility dlllist plugin.
        @see volatility/plugins/taskmods.py
        """
        results = []

        command = self.plugins["dlllist"](self.config)
        for task in command.calculate():
            new = {
                "process_id": int(task.UniqueProcessId),
                "process_name": str(task.ImageFileName),
                "commandline": str(task.Peb.ProcessParameters.CommandLine or ""),
                "loaded_modules": []
            }

            for module in task.get_load_modules():
                new["loaded_modules"].append({
                    "dll_base": str(module.DllBase),
                    "dll_size": str(module.SizeOfImage),
                    "dll_full_name": str(module.FullDllName or ""),
                    "dll_load_count": int(module.LoadCount),
                })

            results.append(new)

        return dict(config={}, data=results)

    def handles(self):
        """Volatility handles plugin.
        @see volatility/plugins/handles.py
        """
        results = []

        command = self.plugins["handles"](self.config)
        for pid, handle, object_type, name in command.calculate():
            new = {
                "process_id": int(pid),
                "handle_value": str(handle.HandleValue),
                "handle_granted_access": str(handle.GrantedAccess),
                "handle_type": str(object_type),
                "handle_name": str(name)
            }

            results.append(new)

        return dict(config={}, data=results)

    def ldrmodules(self):
        """Volatility ldrmodules plugin.
        @see volatility/plugins/malware/malfind.py
        """
        results = []

        command = self.plugins["ldrmodules"](self.config)
        for task in command.calculate():
            # Build a dictionary for all three PEB lists where the
            # keys are base address and module objects are the values.
            inloadorder = dict((mod.DllBase.v(), mod) for mod in task.get_load_modules())
            ininitorder = dict((mod.DllBase.v(), mod) for mod in task.get_init_modules())
            inmemorder = dict((mod.DllBase.v(), mod) for mod in task.get_mem_modules())

            # Build a similar dictionary for the mapped files.
            mapped_files = {}
            for vad, address_space in task.get_vads(vad_filter=task._mapped_file_filter):
                # Note this is a lot faster than acquiring the full
                # vad region and then checking the first two bytes.
                if obj.Object("_IMAGE_DOS_HEADER", offset=vad.Start, vm=address_space).e_magic != 0x5A4D:
                    continue

                mapped_files[int(vad.Start)] = str(vad.FileObject.FileName or "")

            # For each base address with a mapped file, print info on
            # the other PEB lists to spot discrepancies.
            for base in mapped_files.keys():
                # Does the base address exist in the PEB DLL lists?
                load_mod = inloadorder.get(base, None)
                init_mod = ininitorder.get(base, None)
                mem_mod = inmemorder.get(base, None)

                new = {
                    "process_id": int(task.UniqueProcessId),
                    "process_name": str(task.ImageFileName),
                    "dll_base": "{0:#x}".format(base),
                    "dll_in_load": load_mod is not None,
                    "dll_in_init": init_mod is not None,
                    "dll_in_mem": mem_mod is not None,
                    "dll_mapped_path": str(mapped_files[base]),
                    "load_full_dll_name": "",
                    "init_full_dll_name": "",
                    "mem_full_dll_name": ""
                }

                if load_mod:
                    new["load_full_dll_name"] = str(load_mod.FullDllName)

                if init_mod:
                    new["init_full_dll_name"] = str(init_mod.FullDllName)

                if mem_mod:
                    new["mem_full_dll_name"] = str(mem_mod.FullDllName)

                results.append(new)

        return dict(config={}, data=results)

    def mutantscan(self):
        """Volatility mutantscan plugin.
        @see volatility/plugins/filescan.py
        """
        results = []

        command = self.plugins["mutantscan"](self.config)
        for mutant in command.calculate():
            header = mutant.get_object_header()
            tid = 0
            pid = 0
            if mutant.OwnerThread > 0x80000000:
                thread = mutant.OwnerThread.dereference_as("_ETHREAD")
                tid = thread.Cid.UniqueThread
                pid = thread.Cid.UniqueProcess

            new = {
                "mutant_offset": "{0:#x}".format(mutant.obj_offset),
                "num_pointer": int(header.PointerCount),
                "num_handles": int(header.HandleCount),
                "mutant_signal_state": str(mutant.Header.SignalState),
                "mutant_name": str(header.NameInfo.Name or ""),
                "process_id": int(pid),
                "thread_id": int(tid)
            }

            results.append(new)

        return dict(config={}, data=results)

    def devicetree(self):
        """Volatility devicetree plugin.
        @see volatility/plugins/malware/devicetree.py
        """
        results = []

        command = self.plugins["devicetree"](self.config)
        for driver_obj in command.calculate():
            new = {
                "driver_offset": "0x{0:08x}".format(driver_obj.obj_offset),
                "driver_name": str(driver_obj.DriverName or ""),
                "devices": []
            }

            for device in driver_obj.devices():
                device_header = obj.Object(
                    "_OBJECT_HEADER",
                    offset=device.obj_offset - device.obj_vm.profile.get_obj_offset("_OBJECT_HEADER", "Body"),
                    vm=device.obj_vm,
                    native_vm=device.obj_native_vm
                )

                device_name = str(device_header.NameInfo.Name or "")

                new_device = {
                    "device_offset": "0x{0:08x}".format(device.obj_offset),
                    "device_name": device_name,
                    "device_type": devicetree.DEVICE_CODES.get(device.DeviceType.v(), "UNKNOWN"),
                    "devices_attached": []
                }

                new["devices"].append(new_device)

                level = 0

                for att_device in device.attached_devices():
                    device_header = obj.Object(
                        "_OBJECT_HEADER",
                        offset=att_device.obj_offset - att_device.obj_vm.profile.get_obj_offset("_OBJECT_HEADER", "Body"),
                        vm=att_device.obj_vm,
                        native_vm=att_device.obj_native_vm
                    )

                    device_name = str(device_header.NameInfo.Name or "")
                    name = (device_name + " - " + str(att_device.DriverObject.DriverName or ""))

                    new_device["devices_attached"].append({
                        "level": level,
                        "attached_device_offset": "0x{0:08x}".format(att_device.obj_offset),
                        "attached_device_name": name,
                        "attached_device_type": devicetree.DEVICE_CODES.get(att_device.DeviceType.v(), "UNKNOWN")
                    })

                    level += 1

            results.append(new)

        return dict(config={}, data=results)

    def svcscan(self):
        """Volatility svcscan plugin - scans for services.
        @see volatility/plugins/malware/svcscan.py
        """
        results = []

        command = self.plugins["svcscan"](self.config)
        for rec in command.calculate():
            new = {
                "service_offset": "{0:#x}".format(rec.obj_offset),
                "service_order": int(rec.Order),
                "process_id": int(rec.Pid),
                "service_name": str(rec.ServiceName.dereference()),
                "service_display_name": str(rec.DisplayName.dereference()),
                "service_type": str(rec.Type),
                "service_binary_path": str(rec.Binary),
                "service_state": str(rec.State)
            }

            results.append(new)

        return dict(config={}, data=results)

    def modscan(self):
        """Volatility modscan plugin.
        @see volatility/plugins/modscan.py
        """
        results = []

        command = self.plugins["modscan"](self.config)
        for ldr_entry in command.calculate():
            new = {
                "kernel_module_offset": "{0:#x}".format(ldr_entry.obj_offset),
                "kernel_module_name": str(ldr_entry.BaseDllName or ""),
                "kernel_module_file": str(ldr_entry.FullDllName or ""),
                "kernel_module_base": "{0:#x}".format(ldr_entry.DllBase),
                "kernel_module_size": int(ldr_entry.SizeOfImage),
            }

            results.append(new)

        return dict(config={}, data=results)

    def imageinfo(self):
        """Volatility imageinfo plugin.
        @see volatility/plugins/imageinfo.py
        """
        results = []

        command = self.plugins["imageinfo"](self.config)
        new = {}
        for key, value in command.calculate():
            new[key] = value

        osp = new["Suggested Profile(s)"].split(",")[0]
        new["osprofile"] = osp

        results.append(new)

        return dict(config={}, data=results)

    def sockscan(self):
        """Volatility sockscan plugin.
        @see volatility/plugins/sockscan.py
        """
        results = []

        command = self.plugins["sockscan"](self.config)
        for sock in command.calculate():
            new = {
                "offset": "{0:#010x}".format(sock.obj_offset),
                "process_id": str(sock.Pid),
                "address": str(sock.LocalIpAddress),
                "port": str(sock.LocalPort),
                "protocol": "{0} ({1})".format(sock.Protocol, protos.protos.get(sock.Protocol.v(), "-")),
                "create_time": time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(int(sock.CreateTime))),
            }
            results.append(new)

        return dict(config={}, data=results)

    def netscan(self):
        """Volatility sockscan plugin.
        @see volatility/plugins/netscan.py
        """
        results = []

        command = self.plugins["netscan"](self.config)
        for net_obj, proto, laddr, lport, raddr, rport, state in command.calculate():
            new = {
                "offset": "{0:#010x}".format(net_obj.obj_offset),
                "process_id": str(net_obj.Owner.UniqueProcessId),
                "local_address": str(laddr),
                "local_port": str(lport),
                "remote_address": str(raddr),
                "remote_port": str(rport),
                "protocol": str(proto),
            }
            results.append(new)

        return dict(config={}, data=results)

class VolatilityManager(object):
    """Handle several volatility results."""
    PLUGINS = [
        "pslist",
        "psxview",
        "callbacks",
        ["idt", "x86"],
        "ssdt",
        ["gdt", "x86"],
        "timers",
        "messagehooks",
        "getsids",
        "privs",
        "malfind",
        "apihooks",
        "dlllist",
        "handles",
        "ldrmodules",
        "mutantscan",
        "devicetree",
        "svcscan",
        "modscan",
        "yarascan",
        ["sockscan", "winxp"],
        ["netscan", "vista", "win7"],
    ]

    def __init__(self, memfile, osprofile=None):
        self.mask_pid = []
        self.taint_pid = set()
        self.memfile = memfile

        conf_path = os.path.join(CUCKOO_ROOT, "conf", "memory.conf")
        if not os.path.exists(conf_path):
            log.error("Configuration file {0} not found".format(conf_path))
            self.voptions = False
            return

        self.voptions = Config("memory")

        for pid in self.voptions.mask.pid_generic.split(","):
            pid = pid.strip()
            if pid:
                self.mask_pid.append(int(pid))

        self.no_filter = not self.voptions.mask.enabled
        if self.voptions.basic.guest_profile:
            self.osprofile = self.voptions.basic.guest_profile
        else:
            self.osprofile = osprofile or self.get_osprofile()

    def get_osprofile(self):
        """Get the OS profile"""
        return VolatilityAPI(self.memfile).imageinfo()["data"][0]["osprofile"]

    def run(self):
        results = {}

        # Exit if options were not loaded.
        if not self.voptions:
            return

        vol = VolatilityAPI(self.memfile, self.osprofile)

        for plugin_name in self.PLUGINS:
            if isinstance(plugin_name, list):
                plugin_name, profiles = plugin_name[0], plugin_name[1:]
            else:
                profiles = []

            # Some plugins can only run in certain profiles (i.e., only in
            # Windows XP/Vista/7, or only in x86 or x64).
            osp = self.osprofile.lower()
            for profile in profiles:
                if osp.startswith(profile) or osp.endswith(profile):
                    break
            else:
                if profiles:
                    continue

            plugin = self.voptions.get(plugin_name)
            if not plugin or not plugin.enabled:
                log.debug("Skipping '%s' volatility module", plugin_name)
                continue

            if plugin_name in vol.plugins:
                log.debug("Executing volatility '%s' module.", plugin_name)
                results[plugin_name] = getattr(vol, plugin_name)()

        self.find_taint(results)
        self.cleanup()

        return self.mask_filter(results)

    def mask_filter(self, old):
        """Filter out masked stuff. Keep tainted stuff."""
        new = {}

        for akey in old.keys():
            new[akey] = {"config": old[akey]["config"], "data": []}
            conf = getattr(self.voptions, akey, None)
            new[akey]["config"]["filter"] = conf.filter
            for item in old[akey]["data"]:
                # TODO: need to improve this logic.
                if not conf.filter:
                    new[akey]["data"].append(item)
                elif "process_id" in item and \
                        item["process_id"] in self.mask_pid and \
                        item["process_id"] not in self.taint_pid:
                    pass
                else:
                    new[akey]["data"].append(item)
        return new

    def find_taint(self, res):
        """Find tainted items."""
        if "malfind" in res:
            for item in res["malfind"]["data"]:
                self.taint_pid.add(item["process_id"])

    def cleanup(self):
        """Delete the memory dump (if configured to do so)."""

        if self.voptions.basic.delete_memdump:
            try:
                os.remove(self.memfile)
            except OSError:
                log.error("Unable to delete memory dump file at path \"%s\" ", self.memfile)

class Memory(Processing):
    """Volatility Analyzer."""

    def run(self):
        """Run analysis.
        @return: volatility results dict.
        """
        self.key = "memory"

        results = {}
        if HAVE_VOLATILITY:
            if self.memory_path and os.path.exists(self.memory_path):
                try:
                    results = VolatilityManager(self.memory_path).run()
                except Exception:
                    log.exception("Generic error executing volatility")
            else:
                log.error("Memory dump not found: to run volatility you have to enable memory_dump")
        else:
            log.error("Cannot run volatility module: volatility library not available")

        return results


text:
/*
 *	package : test
 *	source  : TestBeanInfo.java
 *	date    : 1999.8.8
 */

package test;

import java.beans.*;
import java.awt.event.*;

public class TestBeanInfo extends SimpleBeanInfo {

	public TestBeanInfo(){
	}

	/**
	 *	bean descriptor
	 */
	public BeanDescriptor getBeanDescriptor(){
		BeanDescriptor bd = new BeanDescriptor(TestBean.class);

		return bd;
	}

	/**
	 *	property descriptor
	 */
	public PropertyDescriptor[] getPropertyDescriptors(){
		try {
			PropertyDescriptor state = new PropertyDescriptor("state", TestBean.class);
			state.setBound(true);
			state.setConstrained(true);

			PropertyDescriptor[] pds = new PropertyDescriptor[]{state};
			return pds;
		}catch(IntrospectionException e){
			return null;
		}

	}

	/**
	 *	event set descriptors
	 */
	public EventSetDescriptor[] getEventSetDescriptors(){
		try{
			String[] names1 = {"actionPerformed"};
			EventSetDescriptor ed1 = new EventSetDescriptor( TestBean.class,
									"TestEvent",
									TestListener.class,
									names1,
									"addTestListener",
									"removeTestListener");

			String[] names2 = {"FocusGrained","FocusLost"};
			EventSetDescriptor ed2 = new EventSetDescriptor( TestBean.class,
									"FocusEvent",
									FocusListener.class,
									names2,
									"addFocusListener",
									"removeFocusListener");

			EventSetDescriptor[] eda = {ed1,ed2};

			return eda;
		}catch(IntrospectionException e){
			return null;
		}
	}
}


text:
package org.fcrepo.server.storage.translation;

import java.io.ByteArrayInputStream;
import java.io.InputStream;

import java.util.Date;

import org.apache.commons.codec.digest.DigestUtils;
import org.custommonkey.xmlunit.XMLTestCase;

import org.jrdf.graph.URIReference;

import org.junit.Before;

import org.fcrepo.common.Constants;
import org.fcrepo.common.PID;

import org.fcrepo.server.Context;
import org.fcrepo.server.errors.StreamIOException;
import org.fcrepo.server.storage.types.BasicDigitalObject;
import org.fcrepo.server.storage.types.DSBinding;
import org.fcrepo.server.storage.types.DSBindingMap;
import org.fcrepo.server.storage.types.DatastreamManagedContent;
import org.fcrepo.server.storage.types.DatastreamReferencedContent;
import org.fcrepo.server.storage.types.DatastreamXMLMetadata;
import org.fcrepo.server.storage.types.DigitalObject;
import org.fcrepo.server.storage.types.Disseminator;

/**
 * Convenience superclass for serializer and deserializer tests.
 *
 * @author Chris Wilper
 */
@SuppressWarnings("deprecation")
public abstract class TranslationTest
        extends XMLTestCase {

    protected static final String TEST_PID = "test:pid";

    //---
    // Setup/Teardown
    //---

    @Override
    @Before
    public void setUp() {
        // HACK: make DOTranslationUtility happy
        System.setProperty("fedoraServerHost", "localhost");
        System.setProperty("fedoraServerPort", "8080");
        System.setProperty("fedoraAppServerContext", Constants.FEDORA_DEFAULT_APP_CONTEXT);
    }

    //---
    // Static helpers
    //---

    protected static DigitalObject createTestObject(URIReference... contentModelURIs) {
        DigitalObject obj = new BasicDigitalObject();
        obj.setPid(TEST_PID);
        DatastreamXMLMetadata ds = createXDatastream("RELS-EXT");

        StringBuilder rdf = new StringBuilder();
        rdf
                .append("<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\" "
                        + "xmlns:fedora-model=\"info:fedora/fedora-system:def/model#\">\n"
                        + "<rdf:Description rdf:about=\"");
        rdf.append(PID.getInstance(TEST_PID).toURI() + "\">\n");

        for (URIReference model : contentModelURIs) {
            rdf.append("<fedora-model:hasModel rdf:resource=\""
                    + model.getURI().toString()
                    + "\"></fedora-model:hasModel>\n");
        }
        rdf.append("</rdf:Description></rdf:RDF>");
        ds.xmlContent = rdf.toString().getBytes();

        obj.addDatastreamVersion(ds, false);
        obj.setCreateDate(new Date());
        obj.setLastModDate(new Date());
        obj.setCreateDate(new Date());
        obj.setLastModDate(new Date());
        return obj;
    }

    protected static DatastreamXMLMetadata createXDatastream(String id) {
        DatastreamXMLMetadata ds = new DatastreamXMLMetadata();
        ds.DatastreamID = id;
        ds.DSVersionID = id + ".0";
        ds.DSControlGrp = "X";
        ds.xmlContent = "<doc/>".getBytes();
        ds.DSCreateDT = new Date();
        return ds;
    }

    protected static DatastreamReferencedContent createRDatastream(String id,
                                                                   String url) {
        DatastreamReferencedContent ds = new DatastreamReferencedContent();
        ds.DatastreamID = id;
        ds.DSVersionID = id + ".0";
        ds.DSControlGrp = "R";
        ds.DSLocation = url;
        return ds;
    }

    protected static DatastreamManagedContent createMDatastream(String id, final byte [] content) {
        DatastreamManagedContent dmc = new DatastreamManagedContent(){
            public InputStream getContentStream(Context ctx) throws StreamIOException {
                return new ByteArrayInputStream(content);
            }
        };
        dmc.DatastreamID = id;
        dmc.DSVersionID = id + ".0";
        dmc.DSControlGrp = "M";
        dmc.DSChecksumType = "MD5";
        dmc.DSChecksum = DigestUtils.md5Hex(content);
        return dmc;
    }

    protected static Disseminator createDisseminator(String id, int numBindings) {
        Disseminator diss = new Disseminator();
        diss.dissID = id;
        diss.dissVersionID = id + ".0";
        diss.bDefID = TEST_PID + "bdef";
        diss.sDepID = TEST_PID + "bmech";
        diss.dsBindMap = new DSBindingMap();
        // the following is only needed for METS
        diss.dsBindMapID = id + "bindMap";
        DSBinding[] dsBindings = new DSBinding[numBindings];
        for (int i = 1; i <= numBindings; i++) {
            dsBindings[i - 1] = new DSBinding();
            dsBindings[i - 1].bindKeyName = "KEY" + i;
            dsBindings[i - 1].datastreamID = "DS" + i;
        }
        diss.dsBindMap.dsBindings = dsBindings;
        return diss;
    }

}


text:
package OptimizationTests.ShortMethodsInliningNonVirtualInvokes.InvokeSuperRangeAObjectThrowNullSet_001;
// The test checks that stack after NullPointerException occurs is correct despite inlining
class Main {
    final static int iterations = 10;

    public static void main(String[] args) {
        Test test = new Test(iterations);

        Foo nextThingy = new Foo();
        Foo g1 = new Foo();
        Foo g2 = new Foo();
        Foo g3 = new Foo();
        Foo g4 = new Foo();
        Foo g5 = new Foo();
        Foo g6 = new Foo();

        for(int i = 0; i < iterations; i++) {
            SuperTest.thingiesArray[i] = new Foo();
        }

        for(int i = 0; i < iterations; i++) {
            nextThingy = test.getThingies(SuperTest.thingiesArray, i, g1, g2, g3, g4, g5, g6);
            if (i == iterations - 1)
                SuperTest.thingiesArray = null;
            test.setThingies(SuperTest.thingiesArray, nextThingy, i, g1, g2, g3, g4, g5, g6);
        }

    }
}


text:
using System.Data.Entity.Migrations;

namespace TestingServerPush.Web.Migrations
{
    public partial class AllModels : DbMigration
    {
        public override void Up()
        {
            CreateTable(
                "dbo.JobActions",
                c => new
                    {
                        Id = c.Int(nullable: false, identity: true),
                        AddedAt = c.DateTime(nullable: false),
                        JobId = c.Int(nullable: false),
                        Name = c.String(nullable: false, maxLength: 255),
                    })
                .PrimaryKey(t => t.Id)
                .ForeignKey("dbo.Jobs", t => t.JobId, cascadeDelete: true)
                .Index(t => t.JobId);

            CreateTable(
                "dbo.Jobs",
                c => new
                    {
                        Id = c.Int(nullable: false, identity: true),
                        AddedAt = c.DateTime(nullable: false),
                        Resolution = c.String(maxLength: 255),
                        StatusId = c.Int(nullable: false),
                        IsInProgress = c.Boolean(nullable: false),
                        Name = c.String(nullable: false, maxLength: 255),
                    })
                .PrimaryKey(t => t.Id)
                .ForeignKey("dbo.Status", t => t.StatusId, cascadeDelete: true)
                .Index(t => t.StatusId);

            CreateTable(
                "dbo.JobPositions",
                c => new
                    {
                        Id = c.Int(nullable: false, identity: true),
                        AddedAt = c.DateTime(nullable: false),
                        JobId = c.Int(nullable: false),
                        Latitude = c.Decimal(nullable: false, precision: 18, scale: 15),
                        Longitude = c.Decimal(nullable: false, precision: 18, scale: 15),
                    })
                .PrimaryKey(t => t.Id)
                .ForeignKey("dbo.Jobs", t => t.JobId, cascadeDelete: true)
                .Index(t => t.JobId);

            CreateTable(
                "dbo.Status",
                c => new
                    {
                        Id = c.Int(nullable: false, identity: true),
                        Name = c.String(nullable: false, maxLength: 255),
                    })
                .PrimaryKey(t => t.Id);

        }

        public override void Down()
        {
            DropForeignKey("dbo.Jobs", "StatusId", "dbo.Status");
            DropForeignKey("dbo.JobPositions", "JobId", "dbo.Jobs");
            DropForeignKey("dbo.JobActions", "JobId", "dbo.Jobs");
            DropIndex("dbo.JobPositions", new[] { "JobId" });
            DropIndex("dbo.Jobs", new[] { "StatusId" });
            DropIndex("dbo.JobActions", new[] { "JobId" });
            DropTable("dbo.Status");
            DropTable("dbo.JobPositions");
            DropTable("dbo.Jobs");
            DropTable("dbo.JobActions");
        }
    }
}


text:
#import <UIKit/UIKit.h>

FOUNDATION_EXPORT NSString * const kAPCActivitiesSectionHeaderViewIdentifier;

@interface APCActivitiesSectionHeaderView : UITableViewHeaderFooterView

@property (weak, nonatomic) IBOutlet UILabel *titleLabel;

@property (weak, nonatomic) IBOutlet UILabel *subTitleLabel;

@end


text:
module NewRelic
  module Agent
    module Instrumentation
      # https://newrelic.com/docs/features/tracking-front-end-time
      # Record queue time metrics based on any of three headers
      # which can be set on the request.
      module QueueTime
        unless defined?(REQUEST_START_HEADER)
          REQUEST_START_HEADER    = 'HTTP_X_REQUEST_START'.freeze
          QUEUE_START_HEADER      = 'HTTP_X_QUEUE_START'.freeze
          MIDDLEWARE_START_HEADER = 'HTTP_X_MIDDLEWARE_START'.freeze
          ALL_QUEUE_METRIC        = 'WebFrontend/QueueTime'.freeze
          # any timestamps before this are thrown out and the parser
          # will try again with a larger unit (2000/1/1 UTC)
          EARLIEST_ACCEPTABLE_TIMESTAMP = 946684800

          CANDIDATE_HEADERS = [
            REQUEST_START_HEADER,
            QUEUE_START_HEADER,
            MIDDLEWARE_START_HEADER
          ].freeze
        end

        module_function

        def parse_frontend_timestamp(headers, now=Time.now)
          earliest = nil

          CANDIDATE_HEADERS.each do |header|
            if headers[header]
              parsed = parse_timestamp(timestamp_string_from_header_value(headers[header]))
              if parsed && (!earliest || parsed < earliest)
                earliest = parsed
              end
            end
          end

          if earliest && earliest > now
            NewRelic::Agent.logger.debug("Negative queue time detected, treating as zero: start=#{earliest.to_f} > now=#{now.to_f}")
            earliest = now
          end

          earliest
        end

        def record_frontend_metrics(start_time, now=Time.now) #THREAD_LOCAL_ACCESS
          NewRelic::Agent.instance.stats_engine.tl_record_unscoped_metrics(
            ALL_QUEUE_METRIC, (now - start_time).to_f)
        end

        def timestamp_string_from_header_value(value)
          case value
          when /^\s*([\d+\.]+)\s*$/ then $1
          # following regexp intentionally unanchored to handle
          # (ie ignore) leading server names
          when /t=([\d+\.]+)/       then $1
          end
        end

        def parse_timestamp(string)
          cut_off = Time.at(EARLIEST_ACCEPTABLE_TIMESTAMP)
          [1_000_000, 1_000, 1].map do |divisor|
            begin
              Time.at(string.to_f / divisor)
            rescue RangeError
              # On Ruby versions built with a 32-bit time_t, attempting to
              # instantiate a Time object in the far future raises a RangeError,
              # in which case we know we've chosen the wrong divisor.
              nil
            end
          end.compact.find { |candidate| candidate > cut_off }
        end
      end
    end
  end
end