text:
A31: gRPC xDS Timeout Support and Config Selector Design
----
* Author(s): Doug Fawley (dfawley)
* Approver: markdroth, ejona86
* Status: Final
* Implemented in:
* Last updated: 2020-07-13
* Discussion at: https://groups.google.com/g/grpc-io/c/iujkovEFSJU

## Abstract

gRPC provides per-method configuration for some features like timeouts and
retries.  This configuration is part of the [Service
Config](https://github.com/grpc/grpc/blob/master/doc/service_config.md),
provided by the Name Resolver, and currently has limited matching semantics
(per-service or per-method with exact name matching only).

In xDS, RDS
[provides](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route_components.proto#route-routeaction)
the ability to configure timeouts and several other features using its matchers,
which support a more rich set of matching semantics than gRPC, including
examining the outgoing HTTP headers and using regexes.

Further complicating matters, gRPC needs to be able to determine timeout
settings before invoking the Load Balancing Policy, which is the point at which
the bulk of the current xDS support is integrated with gRPC.

Extending gRPC to support these matchers directly is not a preferred solution to
the problem, as it could impact performance for all gRPC users, increase our API
surface, and grow our code base for features we otherwise do not intend to
implement.

## Proposal

This proposal is broken up into two sections: the addition of new functionality
to gRPC itself, and then a description of using that functionality to implement
support for xDS RouteAction.

### New Functionality in gRPC

In order to allow the xDS matchers to configure gRPC behavior, but to avoid
implementing support for them directly in gRPC, a new "per-RPC Config Selector"
will be added for internal-only use.  This will be provided by the Name Resolver
component when calling `Update` on the channel in a way that does not extend our
publicly-visible API, similarly to how the xDS Client is provided to the Load
Balancing Policy.

If present, this Config Selector will be invoked once at the beginning of every
RPC (after the name resolver has produced a result).  It will be passed the RPC
method name and the RPC's headers, and it will produce four things:

1.  The same per-RPC configuration which is currently determined by the Service
    Config's MethodConfig.  A default implementation will return the Service
    Config's selected values.
1.  An optional interceptor or callback.  The interceptor would be invoked by
    the channel immediately for the RPC, or the callback would be invoked at
    some point after the channel knows the RPC will no longer call the Load
    Balancing Policy's Pick method and before the end of the RPC.  This is
    necessary for cluster lifecycle management, which will be explained in
    detail later.
1.  Information which will be passed to the Load Balancing Policy's Pick method
    by the channel.  E.g. in Go this could be done via the Context, and Java may
    use CallOptions.  If an interceptor is used for (2) it is possible that the
    interceptor may be capable of satisfying this requirement directly.
1.  An RPC status; if not OK, the RPC is terminated with this status.  This is
    needed in case the RPC cannot be routed.  Like (3), this could be satisfied
    via an interceptor.

Note that, for reasons described below, the Load Balancing Policy's Update
method must be called and return before the Config Selector may be used by the
channel.

### Implementation for xDS RouteAction Support

The "xds" resolver will return a Config Selector which applies the xDS matchers
to the RPC (using the Method name and RPC Header data provided to it) and uses
those inputs to perform the initial routing steps for the RPC based upon the RDS
Routes.  This will enable it to set the timeouts and other settings
appropriately in the MethodConfig according to the RouteAction.  Since the
routing decisions are made here, it will also return the cluster name to the
top-level Load Balancing Policy (described in detail below) in order to inform
the policy which endpoint picker to use for the RPC.  Note that the Config
Selector will also be assuming the responsibility of the current
`weighted_target` Load Balancing Policy.  This means two things: 1. connectivity
state is not considered when making this routing decision, and 2. once a cluster
is chosen for an RPC, retry attempts will only use this cluster.  This matches
Envoy behavior, and also supersedes the behavior specified in [gRFC
A28](https://github.com/grpc/proposal/blob/master/A28-xds-traffic-splitting-and-routing.md).

Because routing is happening in the Config Selector, the "xds_routing"
Load Balancing Policy will no longer be needed.  Instead, the
top-level Load Balancing Policy will be named
"xds_cluster_manager_experimental", and will be a simple aggregator of
child "cds" Load Balancing Policies, although the config format allows
for other types of child Load Balancing Policies for future-proofing.
The Load Balancing Config provided by the Resolver will be a [JSON
representation](developers.google.com/protocol-buffers/docs/proto3#json)
of the following proto message:

```proto
message XDSClusterManagerConfig {
    message Child {
      repeated LoadBalancingConfig child_policy = 1;
    }
    map<string, Child> children = 1;
}
```

The "xds_cluster_manager" policy will maintain a map from cluster name to child
Policy in order to route Picks.

The Resolver and Config Selector need to coordinate removal of clusters when
they are deleted by xDS.  In order to prevent deleted clusters from being
removed from the LB Policy when they may still be needed, the Config Selector
will mark each chosen cluster as in-use and complete at the start and end of
each RPC, respectively.  This could be implemented with atomics, locks, or by
reference-counted objects.  To avoid races, the removal of a cluster will
require two steps:

1.  The Resolver produces a Config Selector incapable of selecting the deleted
    cluster, but a Service Config which still contains it.  This will ensure the
    Load Balancing Policy still contains any cluster chosen by the old Config
    Selector during the transition.
1.  After the Config Selector has been updated and the previous one can no
    longer be called, and when the count of RPCs referencing a deleted cluster
    is zero, the cluster can safely be removed from the Load Balancing Config in
    the Service Config.

This wouldn't need to be encoded as an explicit two-step process -- the first
step should happen as an immediate result of the xDS update, and the second step
can happen when clusters are no longer referenced by anything.  E.g. if the
Config Selector has a reference to the deleted cluster, then when there are no
more references to the config selector, the final reference to the cluster will
be removed, triggering the Service Config update.

Note that when a cluster is deleted in xDS, if the xds_cluster_manager Load
Balancing Policy were to queue RPCs in this scenario, wait-for-ready RPCs would
be stuck indefinitely referencing the cluster, and it could never be removed.
Deleted clusters need to drop the RPC and return an UNAVAILABLE status.

When new clusters are added to xDS, they will be added to both the Config
Selector and Service Config simultaneously.  Here, precautions must be taken to
ensure it is not possible for the Config Selector to be used before the Load
Balancing Policy's Picker has been updated to be aware of the new cluster.  The
channel must update the Load Balancing Policy before applying the Config
Selector, and the Load Balancing Policy must provide a new Picker before its
Update method returns back to the channel. If the Picker does not recognize a
cluster name provided to it, which should not be possible when following the
steps above in that order, it will drop the RPC with an INTERNAL status code.

The following drawing shows which parts of the system will contain which current
Load Balancing Policies.

![xDS Architecture](A31_graphics/xds_architecture.png "xDS Architecture")

#### Supported Fields

Envoy uses two separate timers to monitor streams: one to monitor the entire
duration of the stream and one to monitor the stream after the client indicates
it is done sending.  Because gRPC only uses full-stream timeouts, we will
support fields use to control the first timer, but not the second timer.

Specifically, the following xDS timeout-related settings will be supported:

- [`RouteAction.max_stream_duration.max_stream_duration`](https://github.com/envoyproxy/envoy/blob/6f2ad057f8655e2688a195337f7520dbd77015fa/api/envoy/config/route/v3/route_components.proto#L775)
- [`RouteAction.max_stream_duration.grpc_timeout_header_max`](https://github.com/envoyproxy/envoy/blob/1fb4db754eabb9d659d255d1801b0bfc1a31cd9c/api/envoy/config/route/v3/route_components.proto#L781)
- [`max_stream_duration`](https://github.com/envoyproxy/envoy/blob/6f2ad057f8655e2688a195337f7520dbd77015fa/api/envoy/config/core/v3/protocol.proto#L95) from the [`HttpConnectionManager.common_http_options`](https://github.com/envoyproxy/envoy/blob/6f2ad057f8655e2688a195337f7520dbd77015fa/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#L289) as a default if `RouteAction.max_stream_duration.max_stream_duration` is unset.

gRPC will not support:

- [`RouteAction.timeout`](https://github.com/envoyproxy/envoy/blob/6f2ad057f8655e2688a195337f7520dbd77015fa/api/envoy/config/route/v3/route_components.proto#L952).
  This field begins the timeout counter from a point in time incompatible with gRPC's timeout model.
- [`RouteAction.max_stream_duration.grpc_timeout_header_offset`](https://github.com/envoyproxy/envoy/blob/1fb4db754eabb9d659d255d1801b0bfc1a31cd9c/api/envoy/config/route/v3/route_components.proto#L789).
  This field is used in Envoy for ensuring the proxy detects timeouts before
  the gRPC client cancels the RPC due to its deadline, so supporting it
  directly in the client is unnecessary and would be counter-productive.

If `RouteAction.max_stream_duration.grpc_timeout_header_max` is not set, the
`RouteAction.max_stream_duration.max_stream_duration` field will specify the
maximum RPC timeout for this route (or fall-back to the HTTP Connection Manager
setting).  If it is unset or set to zero, the application's deadline is used
without modification.

If `RouteAction.max_stream_duration.grpc_timeout_header_max` is present, then
`RouteAction.max_stream_duration.max_stream_duration` is ignored and
`RouteAction.max_stream_duration.grpc_timeout_header_max` limits the maximum
timeout for RPCs on this route instead.  A value of 0 indicates the
application's deadline is used without modification.

In all cases, the RPC timeout set by the application may never be exceeded due
to these settings.  For examples, see the following table.  Note that the
`max_stream_duration` column refers to the effective setting based on both
`RouteAction.max_stream_duration.max_stream_duration` and the
`max_stream_duration` from the HTTP Connection Manager's `common_http_options`.

Application Deadline | `grpc_timeout_header_max` | `max_stream_duration` | Effective Timeout
-------------------- | ------------------------- | --------------------- | -----------------
unset | unset | unset | infinite
unset | unset | 0 | infinite
unset | unset | 10s | 10s
unset | 0s | any | infinite
unset | 10s | any | 10s
20s | unset | unset | 20s
20s | unset | 0 | 20s
20s | unset | 10s | 10s
20s | 0s | any | 20s
20s | 10s | any | 10s

## Rationale

### Alternatives Considered

Many alternatives were considered as part of this design.  Below are several of them:

1.  Support xDS matchers in gRPC directly.
    -  This could lead to performance problems for existing users and would
       require extending the gRPC API for features we do not otherwise want.
1.  Create a new "routing" plug-in, similar to the proposal here, but with an
    entry in the service config to configure it.
    -  This means extending our API in ways potentially confusing to our users.
1.  Allow a new class of "scheme" plug-ins that are able to effectively replace
    the Name Resolver, Load Balancing Policy, and client channel implementation.
    -  This would be a significant effort to design and implement.
1.  Allow the Load Balancing Policy to provide an interceptor capable of
    intercepting RPCs early in their lifecycle.
    -  This is similar to the proposed approach, but it would give the power to
       the Load Balancing Policy instead of the Name Resolver.  Since it is
       impacting settings decided before the existing Load Balancing Policy
       integration, this did not seem appropriate.
1.  Name Resolver provides an RPC interceptor.
    -  This option would require more effort than the proposal, since it would
       require implementing retry and hedging support via an interceptor when we
       need to add support for those features.  Note that, since an interceptor
       may be included in this proposal, we have the capability to do similar
       things in the future if needed.


text:
const debug = require('./../debug');

module.exports = (path) => debug(`portal/${path}`);

redpajama_set_name: RedPajamaGithub


text:
from __future__ import unicode_literals

from django.db import models, migrations

class Migration(migrations.Migration):

    dependencies = [
        ('stuff', '0002_authenticator'),
    ]

    operations = [
        migrations.AlterField(
            model_name='authenticator',
            name='user_id',
            field=models.IntegerField(unique=True),
            preserve_default=True,
        ),
    ]


text:
using System;
using System.Globalization;
using System.Threading;
using System.Threading.Tasks;
using Azure;
using Azure.Core;
using Azure.Core.Pipeline;
using Azure.ResourceManager;
using Azure.ResourceManager.Resources;

namespace Azure.ResourceManager.Subscription
{
    /// <summary>
    /// A class representing a collection of <see cref="BillingAccountPolicyResource" /> and their operations.
    /// Each <see cref="BillingAccountPolicyResource" /> in the collection will belong to the same instance of <see cref="TenantResource" />.
    /// To get a <see cref="BillingAccountPolicyCollection" /> instance call the GetBillingAccountPolicies method from an instance of <see cref="TenantResource" />.
    /// </summary>
    public partial class BillingAccountPolicyCollection : ArmCollection
    {
        private readonly ClientDiagnostics _billingAccountPolicyBillingAccountClientDiagnostics;
        private readonly BillingAccountRestOperations _billingAccountPolicyBillingAccountRestClient;

        /// <summary> Initializes a new instance of the <see cref="BillingAccountPolicyCollection"/> class for mocking. </summary>
        protected BillingAccountPolicyCollection()
        {
        }

        /// <summary> Initializes a new instance of the <see cref="BillingAccountPolicyCollection"/> class. </summary>
        /// <param name="client"> The client parameters to use in these operations. </param>
        /// <param name="id"> The identifier of the parent resource that is the target of operations. </param>
        internal BillingAccountPolicyCollection(ArmClient client, ResourceIdentifier id) : base(client, id)
        {
            _billingAccountPolicyBillingAccountClientDiagnostics = new ClientDiagnostics("Azure.ResourceManager.Subscription", BillingAccountPolicyResource.ResourceType.Namespace, Diagnostics);
            TryGetApiVersion(BillingAccountPolicyResource.ResourceType, out string billingAccountPolicyBillingAccountApiVersion);
            _billingAccountPolicyBillingAccountRestClient = new BillingAccountRestOperations(Pipeline, Diagnostics.ApplicationId, Endpoint, billingAccountPolicyBillingAccountApiVersion);
#if DEBUG
			ValidateResourceId(Id);
#endif
        }

        internal static void ValidateResourceId(ResourceIdentifier id)
        {
            if (id.ResourceType != TenantResource.ResourceType)
                throw new ArgumentException(string.Format(CultureInfo.CurrentCulture, "Invalid resource type {0} expected {1}", id.ResourceType, TenantResource.ResourceType), nameof(id));
        }

        /// <summary>
        /// Get Billing Account Policy.
        /// Request Path: /providers/Microsoft.Billing/billingAccounts/{billingAccountId}/providers/Microsoft.Subscription/policies/default
        /// Operation Id: BillingAccount_GetPolicy
        /// </summary>
        /// <param name="billingAccountId"> Billing Account Id. </param>
        /// <param name="cancellationToken"> The cancellation token to use. </param>
        /// <exception cref="ArgumentException"> <paramref name="billingAccountId"/> is an empty string, and was expected to be non-empty. </exception>
        /// <exception cref="ArgumentNullException"> <paramref name="billingAccountId"/> is null. </exception>
        public virtual async Task<Response<BillingAccountPolicyResource>> GetAsync(string billingAccountId, CancellationToken cancellationToken = default)
        {
            Argument.AssertNotNullOrEmpty(billingAccountId, nameof(billingAccountId));

            using var scope = _billingAccountPolicyBillingAccountClientDiagnostics.CreateScope("BillingAccountPolicyCollection.Get");
            scope.Start();
            try
            {
                var response = await _billingAccountPolicyBillingAccountRestClient.GetPolicyAsync(billingAccountId, cancellationToken).ConfigureAwait(false);
                if (response.Value == null)
                    throw new RequestFailedException(response.GetRawResponse());
                return Response.FromValue(new BillingAccountPolicyResource(Client, response.Value), response.GetRawResponse());
            }
            catch (Exception e)
            {
                scope.Failed(e);
                throw;
            }
        }

        /// <summary>
        /// Get Billing Account Policy.
        /// Request Path: /providers/Microsoft.Billing/billingAccounts/{billingAccountId}/providers/Microsoft.Subscription/policies/default
        /// Operation Id: BillingAccount_GetPolicy
        /// </summary>
        /// <param name="billingAccountId"> Billing Account Id. </param>
        /// <param name="cancellationToken"> The cancellation token to use. </param>
        /// <exception cref="ArgumentException"> <paramref name="billingAccountId"/> is an empty string, and was expected to be non-empty. </exception>
        /// <exception cref="ArgumentNullException"> <paramref name="billingAccountId"/> is null. </exception>
        public virtual Response<BillingAccountPolicyResource> Get(string billingAccountId, CancellationToken cancellationToken = default)
        {
            Argument.AssertNotNullOrEmpty(billingAccountId, nameof(billingAccountId));

            using var scope = _billingAccountPolicyBillingAccountClientDiagnostics.CreateScope("BillingAccountPolicyCollection.Get");
            scope.Start();
            try
            {
                var response = _billingAccountPolicyBillingAccountRestClient.GetPolicy(billingAccountId, cancellationToken);
                if (response.Value == null)
                    throw new RequestFailedException(response.GetRawResponse());
                return Response.FromValue(new BillingAccountPolicyResource(Client, response.Value), response.GetRawResponse());
            }
            catch (Exception e)
            {
                scope.Failed(e);
                throw;
            }
        }

        /// <summary>
        /// Checks to see if the resource exists in azure.
        /// Request Path: /providers/Microsoft.Billing/billingAccounts/{billingAccountId}/providers/Microsoft.Subscription/policies/default
        /// Operation Id: BillingAccount_GetPolicy
        /// </summary>
        /// <param name="billingAccountId"> Billing Account Id. </param>
        /// <param name="cancellationToken"> The cancellation token to use. </param>
        /// <exception cref="ArgumentException"> <paramref name="billingAccountId"/> is an empty string, and was expected to be non-empty. </exception>
        /// <exception cref="ArgumentNullException"> <paramref name="billingAccountId"/> is null. </exception>
        public virtual async Task<Response<bool>> ExistsAsync(string billingAccountId, CancellationToken cancellationToken = default)
        {
            Argument.AssertNotNullOrEmpty(billingAccountId, nameof(billingAccountId));

            using var scope = _billingAccountPolicyBillingAccountClientDiagnostics.CreateScope("BillingAccountPolicyCollection.Exists");
            scope.Start();
            try
            {
                var response = await _billingAccountPolicyBillingAccountRestClient.GetPolicyAsync(billingAccountId, cancellationToken: cancellationToken).ConfigureAwait(false);
                return Response.FromValue(response.Value != null, response.GetRawResponse());
            }
            catch (Exception e)
            {
                scope.Failed(e);
                throw;
            }
        }

        /// <summary>
        /// Checks to see if the resource exists in azure.
        /// Request Path: /providers/Microsoft.Billing/billingAccounts/{billingAccountId}/providers/Microsoft.Subscription/policies/default
        /// Operation Id: BillingAccount_GetPolicy
        /// </summary>
        /// <param name="billingAccountId"> Billing Account Id. </param>
        /// <param name="cancellationToken"> The cancellation token to use. </param>
        /// <exception cref="ArgumentException"> <paramref name="billingAccountId"/> is an empty string, and was expected to be non-empty. </exception>
        /// <exception cref="ArgumentNullException"> <paramref name="billingAccountId"/> is null. </exception>
        public virtual Response<bool> Exists(string billingAccountId, CancellationToken cancellationToken = default)
        {
            Argument.AssertNotNullOrEmpty(billingAccountId, nameof(billingAccountId));

            using var scope = _billingAccountPolicyBillingAccountClientDiagnostics.CreateScope("BillingAccountPolicyCollection.Exists");
            scope.Start();
            try
            {
                var response = _billingAccountPolicyBillingAccountRestClient.GetPolicy(billingAccountId, cancellationToken: cancellationToken);
                return Response.FromValue(response.Value != null, response.GetRawResponse());
            }
            catch (Exception e)
            {
                scope.Failed(e);
                throw;
            }
        }
    }
}


text:
require 'core_ext/hash/deep_merge'
require 'core_ext/hash/deep_symbolize_keys'
require 'base64'

# actually, the worker payload can be cleaned up a lot ...

module Travis
  module Build
    class Data
      autoload :Env, 'travis/build/data/env'
      autoload :Var, 'travis/build/data/var'

      DEFAULTS = { }

      DEFAULT_CACHES = {
        apt:       false,
        bundler:   false,
        cocoapods: false,
        composer:  false
      }

      attr_reader :data

      def initialize(data, defaults = {})
        data = data.deep_symbolize_keys
        defaults = defaults.deep_symbolize_keys
        @data = DEFAULTS.deep_merge(defaults.deep_merge(data))
      end

      def urls
        data[:urls] || {}
      end

      def config
        data[:config]
      end

      def hosts
        data[:hosts] || {}
      end

      def paranoid_mode?
        data.fetch(:paranoid, false)
      end

      def skip_resolv_updates?
        data.fetch(:skip_resolv_updates, false)
      end

      def skip_etc_hosts_fix?
        data.fetch(:skip_etc_hosts_fix, false)
      end

      def cache_options
        data[:cache_options] || {}
      end

      def cache(input = config[:cache])
        case input
        when Hash           then input
        when Array          then input.map { |e| cache(e) }.inject(:merge)
        when String, Symbol then { input.to_sym => true }
        when nil            then {} # for ruby 1.9
        when false          then Hash[DEFAULT_CACHES.each_key.with_object(false).to_a]
        else input.to_h
        end
      end

      def cache?(type, default = DEFAULT_CACHES[type])
        type &&= type.to_sym
        !!cache.fetch(type) { default }
      end

      def env_vars
        @env_vars ||= Env.new(self).vars
      end

      def env_vars_groups
        @env_vars_groups ||= Env.new(self).vars_groups
      end

      def raw_env_vars
        data[:env_vars] || []
      end

      class SshKey < Struct.new(:value, :source, :encoded)
        def value
          if encoded?
            Base64.decode64(super)
          else
            super
          end
        end

        def encoded?
          encoded
        end

        def fingerprint
          rsa_key = OpenSSL::PKey::RSA.new(value)
          public_ssh_rsa = "\x00\x00\x00\x07ssh-rsa" + rsa_key.e.to_s(0) + rsa_key.n.to_s(0)
          OpenSSL::Digest::MD5.new(public_ssh_rsa).hexdigest.scan(/../).join(':')
        rescue OpenSSL::PKey::RSAError
        end
      end

      def ssh_key
        if ssh_key = data[:ssh_key]
          SshKey.new(ssh_key[:value], ssh_key[:source], ssh_key[:encoded])
        elsif source_key = data[:config][:source_key]
          SshKey.new(source_key, nil, true)
        end
      end

      def pull_request
        job[:pull_request]
      end

      def secure_env_enabled?
        job[:secure_env_enabled]
      end

      def source_host
        source_url =~ %r(^(?:https?|git)(?:://|@)([^/]*?)(?:/|:)) && $1
      end

      def api_url
        repository[:api_url]
      end

      def source_url
        repository[:source_url]
      end

      def slug
        repository[:slug]
      end

      def commit
        job[:commit]
      end

      def branch
        job[:branch]
      end

      def ref
        job[:ref]
      end

      def job
        data[:job] || {}
      end

      def build
        data[:source] || data[:build] || {} # TODO standarize the payload on :build
      end

      def repository
        data[:repository] || {}
      end

      def token
        data[:oauth_token]
      end
    end
  end
end


text:
import hmac

#Stream

class RC4:
  """Implementation of RC4-DROP-2048 stream cipher"""

  def __init__(self,key):
    self.key = bytearray(key)
    assert len(key)==256
    self.s = [i for i in range(0,256)]
    j = 0
    for i in range(0,256):
      j = (j + self.s[i] + self.key[i % 256]) % 256
      self.s[i], self.s[j] = self.s[j], self.s[i]
    self.getBytes(2048)

  def getBytes(self,bs):
    """Read bytes from the keystream"""
    i = 0
    j = 0
    stream = bytearray()
    for _ in range(bs):
      i = (i + 1) % 256
      j = (j + self.s[i]) % 256
      self.s[i], self.s[j] = self.s[j], self.s[i]
      stream.append(self.s[(self.s[i]+self.s[j]) % 256])
    return stream

  def xor(self,data):
    """XOR a block of data with fresh bytes from the keystream"""
    data = bytearray(data)
    stream = self.getBytes(len(data))
    return bytearray(map(lambda b: b[0]^b[1],zip(data,stream)))

class AES:
  """Use PyCrypto implementation of AES in counter mode"""

  def __init__(self,key):
    from Crypto.Cipher import AES
    self.key = bytearray(key)
    assert len(key)==256
    self.impl = AES.new(key,AES.MODE_CTR)

  def getBytes(self,bs):
    data = bytes(bs)
    return AES.encrypt(data)

  def xor(self,data):
    """XOR (encrypt) a block of data"""
    data = bytearray(data)
    stream = self.getBytes(len(data))
    return bytearray(map(lambda b: b[0]^b[1],zip(data,stream)))

def getBestCipherObject(key):
  try:
    from Crypto.Cipher import AES
    return AES(key)
  except ImportError:
    return RC4(key)

def make_hmacf(key):
  """Build a function for message signing"""
  return (lambda msg: make_hmacf_single_use(key)(msg))

def make_hmacf_single_use(key):
  """Build a function for signing a single message, internal use only"""
  hmac_obj = hmac.new(bytes(key),msg=None,digestmod='sha1')
  return lambda msg: hmac_obj.update(msg) or hmac_obj.digest()


text:
from __future__ import print_function
import csv
import datetime
import os
import random
import re
import string
# from urlparse import urlparse, parse_qs
from locust import HttpLocust, TaskSet, task
from locust.exception import StopLocust
import six
from six.moves.urllib.parse import urlparse, parse_qs

class CASTaskSet(TaskSet):
    execution_pat = re.compile(r'<input type="hidden" name="execution" value="([^"]+)"')
    eventid_pat = re.compile(r'<input type="hidden" name="_eventId" value="([^"]+)"')
    badpasswd_freq = 0.01
    logout_url_freq = 0.25

    def on_start(self):
        """
        Start a new session.
        """
        self.initialize()
        self.reset()

    def initialize(self):
        """
        Initialize parameters for locust that don't change when reset.
        """
        print("Initializing locust ...")
        self.is_authenticated = False
        lifetime_bins = []
        minute = 60
        hour = 3600
        lifetime_bins.extend([60] * 7)
        lifetime_bins.extend([5 * minute] * 2)
        lifetime_bins.extend([2 * hour] * 1)
        seconds = random.choice(lifetime_bins)
        self.base_lifetime = seconds

    def reset(self):
        self.is_authenticated = False
        self.login()
        wiggliness = random.uniform(0.85, 1.15)
        seconds = int(self.base_lifetime * wiggliness)
        self.expiration = datetime.datetime.now() + datetime.timedelta(seconds=seconds)
        print("Locust created.  Expires in {0} seconds.".format(seconds))

    def login(self):
        """
        Login, obtain TGT.
        Simulates a bad password `badpasswd_freq` percent of the time.
        Sets self.is_authenticated if TGT is obtained.
        """
        print("Authenticating ...")
        client = self.client
        with client.get("/cas/login", catch_response=True) as response:
            if response.status_code == 404:
                response.success()
            content = response.content.decode('utf-8')
        m = self.execution_pat.search(content)
        if m is None:
            return
        execution = m.groups()[0]
        m = self.eventid_pat.search(content)
        if m is None:
            return
        event_id = m.groups()[0]
        creds = random.choice(self.locust.creds)
        user = creds[0]
        passwd = creds[1]
        if self.badpasswd_freq >= random.random():
            passwd = passwd[:-1]
            self.simulate_badpasswd(user, passwd, execution, event_id)
            return
        data = {
            "username": user,
            "password": passwd,
            "execution": execution,
            "_eventId": event_id,
            "geolocation": "",
        }
        response = client.post("/cas/login", data=data)
        if 'TGC' in client.cookies.keys():
            self.is_authenticated = True

    def simulate_badpasswd(self, user, passwd, execution, event_id):
        """
        Simulate fat-fingering the password.
        """
        client = self.client
        data = {
            "username": user,
            "password": passwd,
            "execution": execution,
            "_eventId": event_id,
            "geolocation": "",
        }
        with client.post("/cas/login", catch_response=True, data=data) as response:
            if response.status_code == 401:
                response.success()
                print("Simulated password mis-entry.")
            else:
                print("Unexpected successful authentication!")
                response.failure("Got status code {0} when 401 was expected!".format(response.status_code))

    @task
    def authenticate_to_service(self):
        if not self.is_authenticated:
            print("Recycling locust ...")
            self.reset()
            return
        if datetime.datetime.now() >= self.expiration:
            print("Locust is expired.")
            self.expire()
            self.reset()
            return
        service = '''https://badges.stage.lafayette.edu/'''
        client = self.client
        print("Obtaining service ticket ...")
        with client.get("/cas/login", catch_response=True, params={'service': service},
                        allow_redirects=False) as response:
            if response.status_code in (301, 302):
                response.success()
                location = response.headers['Location']
            else:
                response.failure("Got status code {0} while obtaining ST.".format(response.status_code))
                return
        p = urlparse(location)
        q = p.query
        qs = parse_qs(q)
        ticket = qs['ticket']
        print("Validating service ticket ...")
        response = client.get(
            "/cas/serviceValidate",
            params={'service': service, 'ticket': ticket},
            name="/cas/serviceValidate?ticket=[ticket]")

    def expire(self):
        """
        Logout, or simulate just closing the browser window.
        """
        self.is_authenticated = False
        if self.logout_url_freq >= random.random():
            self.logout()
        else:
            self.client.cookies.clear()

    def logout(self):
        """
        Logout.
        This locust should be reset as a new hatchling."
        """
        client = self.client
        client.get("/cas/logout")
        print("Logged out of SSO.")

def load_creds():
    """
    Load test user credentials.
    """
    credpath = os.path.join(
        os.path.dirname(__file__),
        "credentials.csv")
    creds = []
    with open(credpath, "r") as f:
        reader = csv.reader(f)
        for row in reader:
            creds.append((row[0], row[1]))
    return creds

class CASLocust(HttpLocust):
    task_set = CASTaskSet
    host = 'https://cas.stage.lafayette.edu'
    second = 1000
    min_wait = 5 * second
    max_wait = 15 * second
    creds = load_creds()


text:
package com.fishercoder.solutions;

import com.fishercoder.common.classes.TreeNode;

import java.util.ArrayList;
import java.util.List;

public class _257 {
    public static class Solution1 {
        //a very typical/good question to test your recursion/dfs understanding.
        public List<String> binaryTreePaths_more_concise(TreeNode root) {
            List<String> paths = new ArrayList<>();
            if (root == null) {
                return paths;
            }
            dfs(root, paths, "");
            return paths;
        }

        private void dfs(TreeNode root, List<String> paths, String path) {
            if (root.left == null && root.right == null) {
                paths.add(path + root.val);
                return;
            }
            path += root.val + "->";
            if (root.left != null) {
                dfs(root.left, paths, path);
            }
            if (root.right != null) {
                dfs(root.right, paths, path);
            }
        }
    }

    public static class Solution2 {
        public List<String> binaryTreePaths(TreeNode root) {
            List<String> paths = new ArrayList<>();
            dfs(root, paths, new StringBuilder());
            return paths;
        }

        private void dfs(TreeNode root, List<String> paths, StringBuilder sb) {
            if (root == null) {
                return;
            }
            if (root.left == null && root.right == null) {
                sb.append(root.val);
                paths.add(sb.toString());
                return;
            }
            sb.append(root.val + "->");
            String curr = sb.toString();
            if (root.left != null) {
                dfs(root.left, paths, sb);
            }
            sb.setLength(0);
            sb.append(curr);
            if (root.right != null) {
                dfs(root.right, paths, sb);
            }
        }
    }
}


text:
package com.ibm.watson.discovery.v1.model;

import static org.testng.Assert.*;

import com.ibm.cloud.sdk.core.service.model.FileWithMetadata;
import com.ibm.watson.discovery.v1.utils.TestUtilities;
import java.io.InputStream;
import java.util.HashMap;
import java.util.List;
import org.testng.annotations.Test;

/** Unit test class for the QueryNoticesOptions model. */
public class QueryNoticesOptionsTest {
  final HashMap<String, InputStream> mockStreamMap = TestUtilities.createMockStreamMap();
  final List<FileWithMetadata> mockListFileWithMetadata =
      TestUtilities.creatMockListFileWithMetadata();

  @Test
  public void testQueryNoticesOptions() throws Throwable {
    QueryNoticesOptions queryNoticesOptionsModel =
        new QueryNoticesOptions.Builder()
            .environmentId("testString")
            .collectionId("testString")
            .filter("testString")
            .query("testString")
            .naturalLanguageQuery("testString")
            .passages(true)
            .aggregation("testString")
            .count(Long.valueOf("26"))
            .xReturn(java.util.Arrays.asList("testString"))
            .offset(Long.valueOf("26"))
            .sort(java.util.Arrays.asList("testString"))
            .highlight(false)
            .passagesFields(java.util.Arrays.asList("testString"))
            .passagesCount(Long.valueOf("100"))
            .passagesCharacters(Long.valueOf("50"))
            .deduplicateField("testString")
            .similar(false)
            .similarDocumentIds(java.util.Arrays.asList("testString"))
            .similarFields(java.util.Arrays.asList("testString"))
            .build();
    assertEquals(queryNoticesOptionsModel.environmentId(), "testString");
    assertEquals(queryNoticesOptionsModel.collectionId(), "testString");
    assertEquals(queryNoticesOptionsModel.filter(), "testString");
    assertEquals(queryNoticesOptionsModel.query(), "testString");
    assertEquals(queryNoticesOptionsModel.naturalLanguageQuery(), "testString");
    assertEquals(queryNoticesOptionsModel.passages(), Boolean.valueOf(true));
    assertEquals(queryNoticesOptionsModel.aggregation(), "testString");
    assertEquals(queryNoticesOptionsModel.count(), Long.valueOf("26"));
    assertEquals(queryNoticesOptionsModel.xReturn(), java.util.Arrays.asList("testString"));
    assertEquals(queryNoticesOptionsModel.offset(), Long.valueOf("26"));
    assertEquals(queryNoticesOptionsModel.sort(), java.util.Arrays.asList("testString"));
    assertEquals(queryNoticesOptionsModel.highlight(), Boolean.valueOf(false));
    assertEquals(queryNoticesOptionsModel.passagesFields(), java.util.Arrays.asList("testString"));
    assertEquals(queryNoticesOptionsModel.passagesCount(), Long.valueOf("100"));
    assertEquals(queryNoticesOptionsModel.passagesCharacters(), Long.valueOf("50"));
    assertEquals(queryNoticesOptionsModel.deduplicateField(), "testString");
    assertEquals(queryNoticesOptionsModel.similar(), Boolean.valueOf(false));
    assertEquals(
        queryNoticesOptionsModel.similarDocumentIds(), java.util.Arrays.asList("testString"));
    assertEquals(queryNoticesOptionsModel.similarFields(), java.util.Arrays.asList("testString"));
  }

  @Test(expectedExceptions = IllegalArgumentException.class)
  public void testQueryNoticesOptionsError() throws Throwable {
    new QueryNoticesOptions.Builder().build();
  }
}


text:
package org.ontoware.rdf2go.osgi;

import org.ontoware.rdf2go.ModelFactory;
import org.ontoware.rdf2go.RDF2Go;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.InvalidSyntaxException;
import org.osgi.framework.ServiceEvent;
import org.osgi.framework.ServiceListener;
import org.osgi.framework.ServiceReference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Activate the RDF2Go framework. Listens to registered OSGI services and
 * searches for registered {@link ModelFactory} instances. The found factories
 * are compared to a configuration value, if the configured factory is found, it
 * is set as default. This is the configuration value to set:
 * org.ontoware.rdf2go.defaultmodelfactory=(classname of modelfactory)
 *
 * This class has been contributed by Deutsches Forschungszentrum fuer
 * Kuenstliche Intelligenz DFKI GmbH.
 *
 * TODO sauermann why cannot just the implementation bundle register itself? It
 * shouldn't be the APIs job to register implementations.
 *
 * @author sauermann
 */
public class RDF2GoActivator implements BundleActivator, ServiceListener {

	/**
	 * set this variable in the OSGI config to define the default driver for
	 * RDF2Go.
	 */
	public static final String DEFAULTMODELFACTORY_CFG = "org.ontoware.rdf2go.defaultmodelfactory";

	/**
	 * The filter used to search for ModelFactories
	 */
	public static final String MODEL_FACTORY_FILTER = "(objectclass="
	        + ModelFactory.class.getName() + ")";

	private BundleContext bc;

	private String defaultFactoryClassName = null;

	private ServiceReference currentFactorySR;

	private static final Logger log = LoggerFactory.getLogger(RDF2GoActivator.class);

	/**
	 * @see org.osgi.framework.BundleActivator#start(org.osgi.framework.BundleContext)
	 */
	@Override
	public void start(BundleContext context) throws Exception {
		this.bc = context;
		this.defaultFactoryClassName = context.getProperty(DEFAULTMODELFACTORY_CFG);
		this.bc.addServiceListener(this, MODEL_FACTORY_FILTER);
		initalizeListener();
	}

	private void initalizeListener() {
		ServiceReference references[];
		try {
			references = this.bc.getServiceReferences((String)null, MODEL_FACTORY_FILTER);
			for(int i = 0; references != null && i < references.length; i++) {
				this.serviceChanged(new ServiceEvent(ServiceEvent.REGISTERED, references[i]));
			}
		} catch(InvalidSyntaxException e) {
			// this will not happen
			log.error("Syntax error", e);
		}
	}

	/**
	 * @see org.osgi.framework.BundleActivator#stop(org.osgi.framework.BundleContext)
	 *
	 *      TODO leo/antoni document why context is not needed/used for stopping
	 *      the bundle
	 */
	@Override
	public void stop(BundleContext context) throws Exception {
		this.bc = null;
	}

	/**
	 * Listen to changes in the ModelFactories.
	 *
	 * @see org.osgi.framework.ServiceListener#serviceChanged(org.osgi.framework.ServiceEvent)
	 */
	@Override
	public void serviceChanged(ServiceEvent event) {
		switch(event.getType()) {
		case ServiceEvent.REGISTERED:
			handleRegisteredEvent(event);
			break;
		case ServiceEvent.UNREGISTERING:
			handleUnregisteredEvent(event);
			break;
		}
		// We do not care for modified
		// case ServiceEvent.MODIFIED:
	}

	private void handleRegisteredEvent(ServiceEvent event) {
		ServiceReference ref = event.getServiceReference();
		ModelFactory factory = (ModelFactory)this.bc.getService(ref);
		String currClass = getCurrentFactoryClassName();
		String newClass = factory.getClass().getName();
		if(this.currentFactorySR == null) {
			log.debug("Registering " + factory.getClass().getName() + " as default ModelFactory");
			this.currentFactorySR = ref;
			RDF2Go.register(factory);
		} else if(this.defaultFactoryClassName != null
		        && !currClass.equals(this.defaultFactoryClassName)
		        && newClass.equals(this.defaultFactoryClassName)) {
			RDF2Go.register((ModelFactory)null);
			this.bc.ungetService(this.currentFactorySR);

			factory = (ModelFactory)this.bc.getService(ref);
			this.currentFactorySR = ref;
			RDF2Go.register(factory);
			log.debug("RDF2Go uses " + newClass + " as default ModelFactory");
		} else {
			this.bc.ungetService(ref);
		}
	}

	private void handleUnregisteredEvent(ServiceEvent event) {
		ServiceReference ref = event.getServiceReference();
		String currClass = getCurrentFactoryClassName();
		if(ref == this.currentFactorySR) {
			RDF2Go.register((ModelFactory)null);
			this.currentFactorySR = null;
			log.debug("RDF2Go unregistered the ModelFactory " + currClass
			        + " as default ModelFactory. No ModelFactory available now. "
			        + "The Bundle of the ModelFactory was unregistered.");
		}
		this.bc.ungetService(ref);
		initalizeListener();
	}

	private static String getCurrentFactoryClassName() {
		ModelFactory currFactory = null;
		try {
			currFactory = RDF2Go.getModelFactory();
		} catch(RuntimeException e) {
			// do nothing, this means that no factory has been registered
		}
		return (currFactory == null ? null : currFactory.getClass().getName());
	}
}


text:
package org.eclipse.jface.text.source;

import org.eclipse.jface.text.IInformationControlCreator;

/**
 * Extension interface for {@link org.eclipse.jface.text.source.IAnnotationHover} for
 *

 *
providing its own information control creator

 *
providing the range of lines for which the hover for a given line is valid

 *
providing whether the information control can interact with the mouse cursor

 *

 *
 * @see org.eclipse.jface.text.IInformationControlCreator
 * @see org.eclipse.jface.text.source.IAnnotationHover
 * @since 3.0
 */
public interface IAnnotationHoverExtension {

	/**
	 * Returns the hover control creator of this annotation hover.
	 *
	 * @return the hover control creator
	 */
	IInformationControlCreator getHoverControlCreator();

	/**
	 * Returns whether the provided information control can interact with the mouse cursor. I.e. the
	 * hover must implement custom information control management.
	 *
	 * @return true if the mouse cursor can be handled
	 */
	boolean canHandleMouseCursor();

	/**
	 * Returns the object which should be presented in the a
	 * hover popup window. The information is requested based on
	 * the specified line range.
	 *
	 * @param sourceViewer the source viewer this hover is registered with
	 * @param lineRange the line range for which information is requested
	 * @param visibleNumberOfLines the number of visible lines
	 * @return the requested information or null if no such information exists
	 */
	Object getHoverInfo(ISourceViewer sourceViewer, ILineRange lineRange, int visibleNumberOfLines);

	/**
	 * Returns the range of lines that include the given line number for which
	 * the same hover information is valid.
	 *
	 * @param viewer the viewer which the hover is queried for
	 * @param lineNumber the line number of the line for which a hover is displayed for
	 * @return the computed line range or null for no range
	 */
	ILineRange getHoverLineRange(ISourceViewer viewer, int lineNumber);
}


text:
import { resolve } from 'path';
import { makeRe } from 'minimatch';
import ensureArray from './utils/ensureArray';

export default function createFilter ( include, exclude ) {
	include = ensureArray( include ).map( id => resolve( id ) ).map( makeRe );
	exclude = ensureArray( exclude ).map( id => resolve( id ) ).map( makeRe );

	return function ( id ) {
		var included = !include.length;

		include.forEach( pattern => {
			if ( pattern.test( id ) ) included = true;
		});

		exclude.forEach( pattern => {
			if ( pattern.test( id ) ) included = false;
		});

		return included;
	};
}


text:
package org.kairosdb.datastore.h2;

import com.google.common.collect.HashMultimap;
import com.google.common.collect.ImmutableSortedMap;
import com.google.inject.Inject;
import com.google.inject.name.Named;
import com.mchange.v2.c3p0.DataSources;
import org.agileclick.genorm.runtime.GenOrmQueryResultSet;
import org.h2.jdbcx.JdbcDataSource;
import org.kairosdb.core.KairosDataPointFactory;
import org.kairosdb.core.datastore.Datastore;
import org.kairosdb.core.datastore.DatastoreMetricQuery;
import org.kairosdb.core.datastore.QueryCallback;
import org.kairosdb.core.datastore.ServiceKeyStore;
import org.kairosdb.core.datastore.ServiceKeyValue;
import org.kairosdb.core.datastore.TagSet;
import org.kairosdb.core.datastore.TagSetImpl;
import org.kairosdb.core.exception.DatastoreException;
import org.kairosdb.datastore.cassandra.DataPointsRowKey;
import org.kairosdb.datastore.h2.orm.DSEnvelope;
import org.kairosdb.datastore.h2.orm.DataPoint;
import org.kairosdb.datastore.h2.orm.DeleteMetricsQuery;
import org.kairosdb.datastore.h2.orm.GenOrmDataSource;
import org.kairosdb.datastore.h2.orm.InsertDataPointQuery;
import org.kairosdb.datastore.h2.orm.Metric;
import org.kairosdb.datastore.h2.orm.MetricIdResults;
import org.kairosdb.datastore.h2.orm.MetricIdsQuery;
import org.kairosdb.datastore.h2.orm.MetricIdsWithTagsQuery;
import org.kairosdb.datastore.h2.orm.MetricNamesPrefixQuery;
import org.kairosdb.datastore.h2.orm.MetricNamesQuery;
import org.kairosdb.datastore.h2.orm.MetricTag;
import org.kairosdb.datastore.h2.orm.MetricTagValuesQuery;
import org.kairosdb.datastore.h2.orm.ServiceIndex;
import org.kairosdb.datastore.h2.orm.ServiceModification;
import org.kairosdb.datastore.h2.orm.Tag;
import org.kairosdb.datastore.h2.orm.TagNamesQuery;
import org.kairosdb.datastore.h2.orm.TagValuesQuery;
import org.kairosdb.eventbus.FilterEventBus;
import org.kairosdb.eventbus.Publisher;
import org.kairosdb.eventbus.Subscribe;
import org.kairosdb.events.DataPointEvent;
import org.kairosdb.events.RowKeyEvent;
import org.kairosdb.util.KDataInput;
import org.kairosdb.util.KDataOutput;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.sql.DataSource;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.util.ArrayList;
import java.util.Date;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.regex.Pattern;

import static org.kairosdb.core.KairosConfigProperties.QUERIES_REGEX_PREFIX;

public class H2Datastore implements Datastore, ServiceKeyStore
{
	public static final Logger logger = LoggerFactory.getLogger(H2Datastore.class);
	public static final String DATABASE_PATH_PROPERTY = "kairosdb.datastore.h2.database_path";
	private static final long MIN_TIME_VALUE = Long.MIN_VALUE / 1000;
	private static final long MAX_TIME_VALUE = Long.MAX_VALUE;

	private Connection m_holdConnection;  //Connection that holds the database open
	private final KairosDataPointFactory m_dataPointFactory;
	private final Publisher<RowKeyEvent> m_rowKeyPublisher;
	private final String m_regexPrefix;

	@Inject
	public H2Datastore(@Named(DATABASE_PATH_PROPERTY) String dbPath,
			KairosDataPointFactory dataPointFactory,
			FilterEventBus eventBus,
			@Named(QUERIES_REGEX_PREFIX) String regexPrefix) throws DatastoreException
	{
		m_dataPointFactory = dataPointFactory;
		m_rowKeyPublisher = eventBus.createPublisher(RowKeyEvent.class);
		boolean createDB = false;
		m_regexPrefix = regexPrefix;

		File dataDir = new File(dbPath);
		if (!dataDir.exists())
			createDB = true;

		dbPath = dbPath.replace('\\', '/');
		//newer H2 is more strict about relative paths
		String jdbcPath = (dataDir.isAbsolute() || dbPath.startsWith("./") ? "" : "./") + dbPath;
		logger.info("Starting H2 database in " + jdbcPath);

		JdbcDataSource jdbcds = new JdbcDataSource();
		jdbcds.setURL("jdbc:h2:" + jdbcPath + "/kairosdb");
		jdbcds.setUser("sa");

		DataSource ds = jdbcds;

		try
		{

			//Uncomment this line to detect db leaks in H2
			//ds = new LeakDetectorDataSource(ds, 4, 9);
			GenOrmDataSource.setDataSource(new DSEnvelope(DataSources.pooledDataSource(ds)));
		}
		catch (SQLException e)
		{
			e.printStackTrace();
		}

		try
		{
			if (createDB)
				createDatabase(ds);
		}
		catch (SQLException e)
		{
			//TODO
			System.out.println("Oh Crap");
			e.printStackTrace();
		}
		catch (IOException e)
		{
			//TODO
			System.out.println("double oh crap");
			e.printStackTrace();
		}
	}

	private void createDatabase(DataSource ds) throws IOException, SQLException
	{
		logger.info("Creating DB");
		m_holdConnection = ds.getConnection();
		m_holdConnection.setAutoCommit(false);

		StringBuilder sb = new StringBuilder();
		try(InputStreamReader reader = new InputStreamReader(getClass().getClassLoader()
				.getResourceAsStream("create.sql")))
		{

			int ch;
			while ((ch = reader.read()) != -1)
				sb.append((char) ch);
		}

		String[] tableCommands = sb.toString().split(";");

		Statement s = m_holdConnection.createStatement();
		for (String command : tableCommands)
			s.execute(command);

		m_holdConnection.commit();
	}

	/*
	 shutdown is only used by unit tests.
	 */
	public void shutdown()
	{
		try {
			m_holdConnection.createStatement().execute("SHUTDOWN");
		}
		catch (SQLException e) {
			logger.error("Failed shutdown:", e);
		}
	}

	@Override
	public void close()
	{
		try
		{
			if (m_holdConnection != null) {
				m_holdConnection.close();
			}
		}
		catch (SQLException e)
		{
			logger.error("Failed closing last connection:", e);
		}
	}

	@Subscribe
	public synchronized void putDataPoint(DataPointEvent event) throws DatastoreException
	{
		GenOrmDataSource.attachAndBegin();
		try
		{
			ImmutableSortedMap<String, String> tags = event.getTags();
			String metricName = event.getMetricName();
			org.kairosdb.core.DataPoint dataPoint = event.getDataPoint();

			String key = createMetricKey(metricName, tags, dataPoint.getDataStoreDataType());
			Metric m = Metric.factory.findOrCreate(key);
			if (m.isNew())
			{
				m.setName(metricName);
				m.setType(dataPoint.getDataStoreDataType());

				for (String name : tags.keySet())
				{
					String value = tags.get(name);
					Tag.factory.findOrCreate(name, value);
					MetricTag.factory.findOrCreate(key, name, value);
				}

				GenOrmDataSource.flush();
				DataPointsRowKey dataPointsRowKey = new DataPointsRowKey(metricName, "H2",
						0, dataPoint.getDataStoreDataType(), tags);
				m_rowKeyPublisher.post(new RowKeyEvent(metricName, dataPointsRowKey, 0));

			}

			KDataOutput dataOutput = new KDataOutput();
			dataPoint.writeValueToBuffer(dataOutput);

			new InsertDataPointQuery(m.getId(), new Timestamp(dataPoint.getTimestamp()),
					dataOutput.getBytes()).runUpdate();

			GenOrmDataSource.commit();
		}
		catch (IOException e)
		{
			throw new DatastoreException(e);
		}
		finally
		{
			GenOrmDataSource.close();
		}
	}

	@Override
	public Iterable<String> getMetricNames(String prefix)
	{
		List<String> metricNames = new ArrayList<>();
		if (prefix == null)
		{
			MetricNamesQuery query = new MetricNamesQuery();
			MetricNamesQuery.ResultSet results = query.runQuery();

			while (results.next())
			{
				metricNames.add(results.getRecord().getName());
			}

			results.close();
		}
		else
		{
			MetricNamesPrefixQuery query = new MetricNamesPrefixQuery(prefix+"%");
			MetricNamesPrefixQuery.ResultSet results = query.runQuery();

			while (results.next())
			{
				metricNames.add(results.getRecord().getName());
			}

			results.close();
		}

		return (metricNames);
	}

	@Override
	public Iterable<String> getTagNames()
	{
		TagNamesQuery.ResultSet results = new TagNamesQuery().runQuery();

		List<String> tagNames = new ArrayList<>();
		while (results.next())
			tagNames.add(results.getRecord().getName());

		results.close();

		return (tagNames);
	}

	@Override
	public Iterable<String> getTagValues()
	{
		TagValuesQuery.ResultSet results = new TagValuesQuery().runQuery();

		List<String> tagValues = new ArrayList<>();
		while (results.next())
			tagValues.add(results.getRecord().getValue());

		results.close();

		return (tagValues);
	}

	private GenOrmQueryResultSet<? extends MetricIdResults> getMetricIdsForQuery(DatastoreMetricQuery query)
	{
		StringBuilder sb = new StringBuilder();

		GenOrmQueryResultSet<? extends MetricIdResults> idQuery;

		HashMultimap<String, String> filterTags = HashMultimap.create();

		//Check if any of the tags are regex
		for (Map.Entry<String, String> tagPair : query.getTags().entries())
		{
			if (m_regexPrefix.length() != 0 && tagPair.getValue().startsWith(m_regexPrefix))
			{
				String regex = tagPair.getValue().substring(m_regexPrefix.length());

				Pattern pattern = Pattern.compile(regex);

				MetricTagValuesQuery.ResultSet resultSet = (new MetricTagValuesQuery(query.getName(), tagPair.getKey())).runQuery();
				while (resultSet.next())
				{
					String tagValue = resultSet.getRecord().getTagValue();

					if (pattern.matcher(tagValue).matches())
						filterTags.put(tagPair.getKey(), tagValue);
				}
				resultSet.close();

			}
			else
			{
				filterTags.put(tagPair.getKey(), tagPair.getValue());
			}
		}

		//todo query tags and values and run the regex over them placing results in to filterTags

		//Manually build the where clause for the tags
		//This is subject to sql injection
		Set<String> filterTagNames = filterTags.keySet();
		if (filterTagNames.size() != 0)
		{
			sb.append(" and (");
			boolean first = true;
			for (String tag : filterTagNames)
			{
				if (!first)
					sb.append(" or ");
				first = false;

				sb.append(" (mt.\"tag_name\" = '").append(tag);
				sb.append("' and (");

				Set<String> values = filterTags.get(tag);
				boolean firstValue = true;
				for (String value : values)
				{
					if (!firstValue)
						sb.append(" or ");
					firstValue = false;

					sb.append("mt.\"tag_value\" = '").append(value);
					sb.append("' ");
				}

				sb.append(")) ");
			}

			sb.append(") ");

			idQuery = new MetricIdsWithTagsQuery(query.getName(), filterTagNames.size(),
					sb.toString()).runQuery();
		}
		else
		{
			idQuery = new MetricIdsQuery(query.getName()).runQuery();
		}

		return (idQuery);
	}

	@Override
	public void queryDatabase(DatastoreMetricQuery query, QueryCallback queryCallback) throws DatastoreException
	{
		GenOrmQueryResultSet<? extends MetricIdResults> idQuery = getMetricIdsForQuery(query);

		try
		{
			while (idQuery.next())
			{
				MetricIdResults result = idQuery.getRecord();

				String metricId = result.getMetricId();
				String type = result.getType();

				//Collect the tags in the results
				MetricTag.ResultSet tags = MetricTag.factory.getByMetric(metricId);
				SortedMap<String, String> tagMap = new TreeMap<>();

				while (tags.next())
				{
					MetricTag mtag = tags.getRecord();
					tagMap.put(mtag.getTagName(), mtag.getTagValue());
				}
				tags.close();

				Timestamp startTime = new Timestamp(query.getStartTime());
				Timestamp endTime = new Timestamp(query.getEndTime());

				DataPoint.ResultSet resultSet;
				if (query.getLimit() == 0)
				{
					resultSet = DataPoint.factory.getForMetricId(metricId,
							startTime, endTime, query.getOrder().getText());
				}
				else
				{
					resultSet = DataPoint.factory.getForMetricIdWithLimit(metricId,
							startTime, endTime, query.getLimit(), query.getOrder().getText());
				}

				try
				{
					if (resultSet.next())
					{
						try (QueryCallback.DataPointWriter dataPointWriter =
								     queryCallback.startDataPointSet(type, tagMap))
						{
							do
							{
								DataPoint record = resultSet.getRecord();

								dataPointWriter.addDataPoint(m_dataPointFactory.createDataPoint(type,
										record.getTimestamp().getTime(),
										KDataInput.createInput(record.getValue())));
							} while (resultSet.next());
						}
					}
				}
				finally
				{
					resultSet.close();
				}
			}
		}
		catch (IOException e)
		{
			throw new DatastoreException(e);
		}
		finally
		{
			idQuery.close();
		}
	}

	@Override
	public void deleteDataPoints(DatastoreMetricQuery deleteQuery) throws DatastoreException
	{
		GenOrmDataSource.attachAndBegin();
		try
		{
			GenOrmQueryResultSet<? extends MetricIdResults> idQuery =
					getMetricIdsForQuery(deleteQuery);

			while (idQuery.next())
			{
				String metricId = idQuery.getRecord().getMetricId();

				new DeleteMetricsQuery(metricId,
						new Timestamp(deleteQuery.getStartTime()),
						new Timestamp(deleteQuery.getEndTime())).runUpdate();

				if (DataPoint.factory.getWithMetricId(metricId) == null)
				{
					Metric.factory.find(metricId).delete();
				}
			}

			idQuery.close();

			GenOrmDataSource.commit();
		}
		finally
		{
			GenOrmDataSource.close();
		}
	}

	@Override
	public TagSet queryMetricTags(DatastoreMetricQuery query) throws DatastoreException
	{
		GenOrmQueryResultSet<? extends MetricIdResults> idQuery = getMetricIdsForQuery(query);

		TagSetImpl tagSet = new TagSetImpl();
		try
		{
			while (idQuery.next())
			{
				String metricId = idQuery.getRecord().getMetricId();

				//Collect the tags in the results
				MetricTag.ResultSet tags = MetricTag.factory.getByMetric(metricId);

				while (tags.next())
				{
					MetricTag mtag = tags.getRecord();
					tagSet.addTag(mtag.getTagName(), mtag.getTagValue());
				}

				tags.close();
			}
		}
		finally
		{
			idQuery.close();
		}

		return tagSet;
	}

	@Override
	public void indexMetricTags(DatastoreMetricQuery query) throws DatastoreException
	{
		// H2 does not have an index
	}

	@Override
	public long getMinTimeValue()
	{
		return MIN_TIME_VALUE;
	}

	@Override
	public long getMaxTimeValue()
	{
		return MAX_TIME_VALUE;
	}

	@Override
	public void setValue(String service, String serviceKey, String key, String value) throws DatastoreException
	{
		GenOrmDataSource.attachAndBegin();
		try
		{
			ServiceIndex serviceIndex = ServiceIndex.factory.findOrCreate(service, serviceKey, key);
			if (value != null) {
				serviceIndex.setValue(value);
				long now = System.currentTimeMillis();
				//Need to make sure the modification time always gets updated
				serviceIndex.setModificationTime(new java.sql.Timestamp(now));

				// Update the service key timestamp
				ServiceModification orCreate = ServiceModification.factory.findOrCreate(service, serviceKey);
				orCreate.setModificationTime(new java.sql.Timestamp(now));
			}

			GenOrmDataSource.commit();
		}
		finally
		{
			GenOrmDataSource.close();
		}
	}

	@Override
	public ServiceKeyValue getValue(String service, String serviceKey, String key) throws DatastoreException
	{
		ServiceIndex serviceIndex = ServiceIndex.factory.find(service, serviceKey, key);
		if (serviceIndex != null)
			return new ServiceKeyValue(serviceIndex.getValue(), serviceIndex.getModificationTime());
		else
			return null;
	}

	@Override
	public Iterable<String> listServiceKeys(String service) throws DatastoreException
	{
		final Iterator<ServiceIndex> keys = ServiceIndex.factory.getServiceKeys(service).getArrayList().iterator();

		return new Iterable<String>()
		{
			@Override
			public Iterator<String> iterator()
			{
				return new Iterator<String>()
				{
					@Override
					public boolean hasNext()
					{
						return keys.hasNext();
					}

					@Override
					public String next()
					{
						return keys.next().getServiceKey();
					}

					@Override
					public void remove() { }
				};
			}
		};
	}

	@Override
	public Iterable<String> listKeys(String service, String serviceKey) throws DatastoreException
	{
		final Iterator<ServiceIndex> keys = ServiceIndex.factory.getKeys(service, serviceKey).getArrayList().iterator();

		return new Iterable<String>()
		{
			@Override
			public Iterator<String> iterator()
			{
				return new Iterator<String>()
				{
					@Override
					public boolean hasNext()
					{
						return keys.hasNext();
					}

					@Override
					public String next()
					{
						return keys.next().getKey();
					}

					@Override
					public void remove() { }
				};
			}
		};
	}

	@Override
	public Iterable<String> listKeys(String service, String serviceKey, String keyStartsWith) throws DatastoreException
	{
		final Iterator<ServiceIndex> results = ServiceIndex.factory.getKeysLike(service, serviceKey, keyStartsWith+"%").getArrayList().iterator();

		return new Iterable<String>()
		{
			@Override
			public Iterator<String> iterator()
			{
				return new Iterator<String>()
				{
					@Override
					public boolean hasNext()
					{
						return results.hasNext();
					}

					@Override
					public String next()
					{
						return results.next().getKey();
					}

					@Override
					public void remove() { }
				};
			}
		};
	}

    @Override
    public void deleteKey(String service, String serviceKey, String key)
            throws DatastoreException
    {
        GenOrmDataSource.attachAndBegin();
        try
        {
            ServiceIndex.factory.delete(service, serviceKey, key);

            // Update the service key timestamp
			ServiceModification orCreate = ServiceModification.factory.findOrCreate(service, serviceKey);
			orCreate.setModificationTime(new java.sql.Timestamp(System.currentTimeMillis()));

			GenOrmDataSource.commit();
        }
        finally
        {
            GenOrmDataSource.close();
        }
    }

	@Override
	public Date getServiceKeyLastModifiedTime(String service, String serviceKey)
			throws DatastoreException
	{
		ServiceModification serviceModification = ServiceModification.factory.find(service, serviceKey);
		if (serviceModification != null)
			return serviceModification.getModificationTime();
		else
			return null;
	}

	private String createMetricKey(String metricName, SortedMap<String, String> tags,
			String type)
	{
		StringBuilder sb = new StringBuilder();
		sb.append(metricName).append(":");

		sb.append(type).append(":");

		for (String name : tags.keySet())
		{
			sb.append(name).append("=");
			sb.append(tags.get(name)).append(":");
		}

		return (sb.toString());
	}
}


text:
<?php
namespace org\shypl\sna;

use org\shypl\common\net\HttpRequest;

interface Adapter {
	/**
	 * @return SocialNetwork
	 */
	public function getNetwork();

	/**
	 * @param string $method
	 * @param array  $parameters
	 *
	 * @return mixed
	 */
	public function callApi($method, array $parameters = array());

	/**
	 * @param HttpRequest $request
	 *
	 * @return UserSession
	 */
	public function authorizeRequest(HttpRequest $request);

	/**
	 * @param PaymentProcessorDelegate $delegate
	 *
	 * @return PaymentProcessor
	 */
	public function createPaymentProcessor(PaymentProcessorDelegate $delegate);

	/**
	 * @param HttpRequest $request
	 *
	 * @return bool
	 */
	public function validateRequest(HttpRequest $request);
}


text:
model small
dosseg

INTNUM      equ 0Ch          ; COM1; COM2: 0Bh
OFFMASK     equ 00010000b    ; COM1; COM2: 00001000b
ONMASK      equ not OFFMASK
UART_BASE   equ 3F8h         ; COM1; COM2: 2F8h
UART_RATE   equ 12           ; 9600 bps, see table in this file
UART_PARAMS equ 00000011b    ; 8n1, see tables
RXFIFOSIZE  equ 8096         ; set this to your needs
TXFIFOSIZE  equ 8096         ; dito.
                             ; the fifos must be large on slow computers
                             ; and can be small on fast ones
                             ; These have nothing to do with the 16550A's
                             ; built-in FIFOs!

.data
long_text  db  0dh
    db  "This is a very long test string. It serves the purpose of",0dh
    db  "demonstrating that our interrupt-driven routines are capable",0dh
    db  "of coping with pressure situations like the one we provoke",0dh
    db  "by sending large bunches of characters in each direction at",0dh
    db  "the same time. Run this test by pressing F1 at a low data",0dh
    db  "rate and a high data rate to see why serial transmission and",0dh
    db  "reception should be programmed interrupt-driven. You won't lose",0dh
    db  "a single character as long as you don't overload the fifos, no",0dh
    db  "matter how hard you try!",0dh,0

ds_dgroup  macro
  mov  ax,DGROUP
  mov  ds,ax
  assume  ds:DGROUP
endm

ds_text  macro
  push  cs
  pop   ds
  assume  ds:_TEXT
endm

rx_checkwrap  macro
  local rx_nowrap
  cmp  si,offset rxfifo+RXFIFOSIZE
  jb  rx_nowrap
  lea  si,rxfifo
rx_nowrap:
endm

tx_checkwrap  macro
  local tx_nowrap
  cmp  si,offset txfifo+TXFIFOSIZE
  jb  tx_nowrap
  lea  si,txfifo
tx_nowrap:
endm

.stack 256

.data?
old_intptr  dd  ?
rxhead      dw  ?
rxtail      dw  ?
txhead      dw  ?
txtail      dw  ?
bitxfifo    dw  1  ; size of built-in TX fifo (1 if no fifo)
rxfifo      db  RXFIFOSIZE dup (?)
txfifo      db  TXFIFOSIZE dup (?)

.code
start  proc far
  call  install_interrupt_handler
  call  clear_fifos
  call  clear_screen
  call  init_UART
continue:
  call  read_RX_fifo
  call  read_keyboard
  jnc  continue
  call  clean_up
  mov  ax,4c00h
  int  21h  ; return to DOS
start  endp

interrupt_handler  proc far
  assume  ds:nothing,es:nothing,ss:nothing,cs:_text
  push  ax
  push  cx
  push  dx  ; first save the regs we need to change
  push  ds
  push  si
  in  al,21h
  or  al,OFFMASK   ; disarm the interrupt
  out 21h,al
  mov  al,20h  ; acknowledge interrupt
  out  20h,al

ih_continue:
  mov  dx,UART_BASE+2
  xor  ax,ax
  in  al,dx  ; get interrupt cause
  test  al,1  ; did the UART generate the int?
  jne  ih_sep  ; no, then it's somebody else's problem
  and  al,6  ; mask bits not needed
  mov  si,ax  ; make a pointer out of it
  call  interrupt_table[si]  ; serve this int
  jmp  ih_continue  ; and look for more things to be done
ih_sep:

  in  al,21h
  and al,ONMASK  ; rearm the interrupt
  out 21h,al

  pop  si
  pop  ds
  pop  dx  ; restore regs
  pop  cx
  pop  ax
  iret
interrupt_table  dw  int_modem,int_tx,int_rx,int_status
interrupt_handler  endp

int_modem  proc near
  ; just clear modem status, we are not interested in it
  mov  dx,UART_BASE+6
  in  al,dx
  ret
int_modem  endp

int_tx  proc near
  ds_dgroup
  ; check if there's something to be sent
  mov  si,txtail
  mov  cx,bitxfifo
itx_more:
  cmp  si,txhead
  je  itx_nothing
  cld
  lodsb
  mov  dx,UART_BASE
  out  dx,al  ; write it to the THR
  ; check for wrap-around in our fifo
  tx_checkwrap
  ; send as much bytes as the chip can take when available
  loop itx_more
  jmp  itx_dontstop
itx_nothing:
  ; no more data in the fifo, so inhibit TX interrupts
  mov  dx,UART_BASE+1
  mov  al,00000001b
  out  dx,al
itx_dontstop:
  mov  txtail,si
  ret
int_tx  endp

int_rx  proc near
  ds_dgroup
  mov  si,rxhead
irx_more:
  mov  dx,UART_BASE
  in  al,dx
  mov  byte ptr [si],al
  inc  si
  ; check for wrap-around
  rx_checkwrap
  ; see if there are more bytes to be read
  mov  dx,UART_BASE+5
  in  al,dx
  test  al,1
  jne  irx_more
  mov  rxhead,si
  test  al,40h  ; Sometimes when sending and receiving at the
  jne  int_tx   ; same time, TX ints get lost. This is a cure.
  ret
int_rx  endp

int_status  proc near
  ; just clear the status ("this trivial task is left as an exercise
  ; to the student")
  mov  dx,UART_BASE+5
  in  al,dx
  ret
int_status  endp

read_RX_fifo  proc near
  ; see if there are bytes to be read from the fifo
  ; we read a maximum of 16 bytes, then return in order
  ; not to break keyboard control
  ds_dgroup
  cld
  mov  cx,16
  mov  si,rxtail
rx_more:
  cmp  si,rxhead
  je  rx_nodata
  lodsb
  call  output_char
  ; check for wrap-around
  rx_checkwrap
  loop  rx_more
rx_nodata:
  mov  rxtail,si
  ret
read_RX_fifo  endp

read_keyboard  proc near
  ds_dgroup
  ; check for keys pressed
  mov  ah,1
  int  16h
  je  rk_nokey
  xor  ax,ax
  int  16h
  cmp  ax,2d18h  ; is it Ctrl-X?
  stc
  je  rk_ctrlx
  cmp  ax,3b00h  ; is it F1?
  jne  rk_nf1
  lea  si,long_text  ; send a very long test string
  call  send_string
  jmp  rk_nokey
rk_nf1:
  ; echo the character to the screen
  call  output_char

  call  send_char
rk_nokey:
  clc
rk_ctrlx:
  ret
read_keyboard  endp

install_interrupt_handler  proc near
  ds_dgroup
  ; install interrupt handler first
  mov  ax,3500h+INTNUM
  int  21h
  mov  word ptr old_intptr,bx
  mov  word ptr old_intptr+2,es
  mov  ax,2500h+INTNUM
  ds_text
  lea  dx,interrupt_handler
  int  21h
  ret
install_interrupt_handler  endp

clear_fifos  proc near
  ds_dgroup
  ; clear fifos (not those in the 16550A, but ours)
  lea  ax,rxfifo
  mov  rxhead,ax
  mov  rxtail,ax
  lea  ax,txfifo
  mov  txhead,ax
  mov  txtail,ax
  ret
clear_fifos  endp

init_UART  proc near
  ; initialize the UART
  mov  dx,UART_BASE+3
  mov  al,80h
  out  dx,al  ; make DL register accessible
  mov  dx,UART_BASE
  mov  ax,UART_RATE
  out  dx,ax  ; write bps rate divisor
  mov  dx,UART_BASE+3
  mov  al,UART_PARAMS
  out  dx,al  ; write parameters

  ; is it a 16550A?
  mov  dx,UART_BASE+2
  in   al,dx
  and  al,11000000b
  cmp  al,11000000b
  jne  iu_nofifos
  mov  bitxfifo,16
  mov  dx,UART_BASE+2
  mov  al,11000111b
  out  dx,al  ; clear and enable the fifos if they exist
iu_nofifos:
  mov  dx,UART_BASE+1
  mov  al,00000001b  ; allow RX interrupts
  out  dx,al
  mov  dx,UART_BASE
  in  al,dx  ; clear receiver
  mov  dx,UART_BASE+5
  in  al,dx  ; clear line status
  inc  dx
  in  al,dx  ; clear modem status
  ; free interrupt in the ICU
  in  al,21h
  and  al,ONMASK
  out  21h,al
  ; and enable ints from the UART
  mov  dx,UART_BASE+4
  mov  al,00001000b
  out  dx,al
  ret
init_UART  endp

clear_screen  proc near
  mov  ah,0fh  ; allow all kinds of video adapters to be used
  int  10h
  cmp  al,7
  je  cs_1
  mov  al,3
cs_1:
  xor  ah,ah
  int  10h
  ret
clear_screen  endp

clean_up  proc near
  ds_dgroup
  ; lock int in the ICU
  in  al,21h
  or  al,OFFMASK
  out  21h,al
  xor  ax,ax
  mov  dx,UART_BASE+4  ; disconnect the UART from the int line
  out  dx,al
  mov  dx,UART_BASE+1  ; disable UART ints
  out  dx,al
  mov  dx,UART_BASE+2  ; disable the fifos (old software relies on it)
  out  dx,al
  ; restore int vector
  lds  dx,old_intptr
  mov  ax,2500h+INTNUM
  int  21h
  ret
clean_up  endp

output_char  proc near
  push  si
  push  ax
oc_cr:
  push  ax
  mov  ah,0eh  ; output character using BIOS TTY
  int  10h     ; it's your task to improve this
  pop  ax
  cmp  al,0dh  ; add LF after CR; change it if you don't like it
  mov  al,0ah
  je  oc_cr
  pop  ax
  pop  si
  ret
output_char  endp

send_char proc near
  push  si
  push  ax
  ds_dgroup
  pop  ax
  mov  si,txhead
  mov  byte ptr [si],al
  inc  si
  ; check for wrap-around
  tx_checkwrap
  mov  txhead,si
  ; test if the interrupt is running at the moment
  mov  dx,UART_BASE+5
  in  al,dx
  test  al,40h
  je  sc_dontcrank
  ; crank it up
  ; note that this might not work with some very old 8250s
  mov  dx,UART_BASE+1
  mov  al,00000011b
  out  dx,al
sc_dontcrank:
  pop  si
  ret
send_char  endp

send_string  proc near
  ; sends a null-terminated string pointed at by DS:SI
  ds_dgroup
  cld
ss_more:
  lodsb
  or  al,al
  je  ss_end
  call send_char
  jmp  ss_more
ss_end:
  ret
send_string  endp

end start


text:
#pragma once

#ifndef MEXPLUS_EOS_TYPES_HPP_
#define MEXPLUS_EOS_TYPES_HPP_

#include "eos/core/Mesh.hpp"
#include "eos/fitting/RenderingParameters.hpp"

#include "mexplus/mxarray.h"

#include "glm/vec2.hpp"
#include "glm/vec3.hpp"
#include "glm/vec4.hpp"
#include "glm/gtc/quaternion.hpp"
#include "glm/mat4x4.hpp"

#include "Eigen/Core"

#include "mex.h"

#include <cstdint>
#include <array>
#include <vector>

/**
 * @file
 * @brief Contains mexplus template specialisations to convert eos data
 * structures into Matlab.
 *
 * Note 1: These all copy the data, which I believe might be necessary, since
 * Matlab may unload a mex module (with all its allocated data) at any given
 * time.
 * Note 2: They all return double vectors and matrices, even when the data given
 * are floats. We can think about changing that if it's a speed issue, however,
 * I think double is Matlab's default data type.
 */

namespace mexplus {

/**
 * @brief Converts a glm::tquat<float> to a Matlab vector.
 *
 * @param[in] quat The quaternion to convert.
 * @return An 1x4 Matlab vector.
 */
template<>
mxArray* MxArray::from(const glm::tquat<float>& quat)
{
	MxArray out_array(MxArray::Numeric<double>(1, 4));
	for (int c = 0; c < 4; ++c) {
		out_array.set(c, quat[c]);
	}
	return out_array.release();
};

/**
 * @brief Converts a glm::tmat4x4<float> to a Matlab (double) matrix.
 *
 * @param[in] mat The matrix to convert.
 * @return A 4x4 Matlab matrix.
 */
template<>
mxArray* MxArray::from(const glm::tmat4x4<float>& mat)
{
	MxArray out_array(MxArray::Numeric<double>(4, 4));
	for (int r = 0; r < 4; ++r) {
		for (int c = 0; c < 4; ++c) {
			out_array.set(r, c, mat[c][r]);
		}
	}
	return out_array.release();
};

/**
 * @brief Converts a 4x4 Matlab matrix to a glm::tmat4x4<float>.
 *
 * @param[in] in_array The matrix data from Matlab.
 * @param[in,out] mat The converted matrix in C++.
 */
template <>
void MxArray::to(const mxArray* in_array, glm::tmat4x4<float>* mat)
{
    MxArray arr(in_array);
    if (!arr.isDouble())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array should be of type double.");
    }
    if (arr.rows() != 4 || arr.cols() != 4)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 4 rows and 4 cols.");
    }

    for (int c = 0; c < 4; ++c) {
        for (int r = 0; r < 4; ++r) {
            (*mat)[c][r] = arr.at<double>(mwIndex(r), mwIndex(c));
        }
    }
};

/**
 * @brief Converts an std::vector of glm::tvec2<float> to a Matlab matrix.
 *
 * This function converts a whole vector of vec2's to an n x 2 Matlab matrix,
 * where n is data.size(). It is mainly used to pass texture coordinates of
 * a Mesh to Matlab.
 *
 * We specialise for std::vector<glm::tvec2<float>> directly (and not for
 * glm::tvec2<float>) because otherwise a cell array of vec2's would be
 * generated. Luckily, even if a tvec2 specialisation was to exist too,
 * this one would take precedence to convert a vector<tvec2>.
 *
 * @param[in] data The data to convert.
 * @return An n x 2 Matlab matrix.
 */
template<>
mxArray* MxArray::from(const std::vector<glm::tvec2<float>>& data)
{
	MxArray out_array(MxArray::Numeric<double>(data.size(), 2));
	for (int r = 0; r < data.size(); ++r) {
		for (int c = 0; c < 2; ++c) {
			out_array.set(r, c, data[r][c]);
		}
	}
	return out_array.release();
};

/**
 * @brief Converts a Matlab matrix into an std::vector of glm::vec2.
 *
 * This function converts an n x 2 Matlab matrix to a vector of vec2's.
 * It is mainly used to pass vertex texture coordinate data from Matlab back to C++.
 *
 * See template <> mxArray* MxArray::from(const std::vector<glm::tvec2<float>>&)
 * for more details regarding the type specialisation.
 *
 * @param[in] in_array The matrix data from Matlab.
 * @param[in,out] data The converted data in C++.
 */
template <>
void MxArray::to(const mxArray* in_array, std::vector<glm::vec2>* data)
{
    MxArray arr(in_array);

    if (arr.dimensionSize() != 2)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 2 dimensions, i.e. n x 2.");
    }
    if (arr.cols() != 2)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 2 elements in the second dimension, i.e. n x 2.");
    }
    if (!arr.isDouble())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array should be of type double.");
    }

    const auto num_vertices = arr.rows();
    data->reserve(num_vertices);

    for (int i = 0; i < num_vertices; ++i)
    {
        data->push_back(
            glm::vec2(arr.at<double>(mwIndex(i), mwIndex(0)), arr.at<double>(mwIndex(i), mwIndex(1))));
    }
};

/**
 * @brief Converts an std::vector of glm::tvec3<float> to a Matlab matrix.
 *
 * This function converts a whole vector of vec3's to an n x 3 Matlab matrix,
 * where n is data.size(). It is mainly used to pass vertex colour data of
 * a Mesh to Matlab.
 *
 * See template <> mxArray* MxArray::from(const std::vector<glm::tvec2<float>>&)
 * for more details.
 *
 * @param[in] data The data to convert.
 * @return An n x 3 Matlab matrix.
 */
template<>
mxArray* MxArray::from(const std::vector<glm::tvec3<float>>& data)
{
	MxArray out_array(MxArray::Numeric<double>(data.size(), 3));
	for (int r = 0; r < data.size(); ++r) {
		for (int c = 0; c < 3; ++c) {
			out_array.set(r, c, data[r][c]);
		}
	}
	return out_array.release();
};

/**
 * @brief Converts an std::vector of glm::tvec4<float> to a Matlab matrix.
 *
 * This function converts a whole vector of vec4's to an n x 4 Matlab matrix,
 * where n is data.size(). It is mainly used to pass vertex data of a Mesh
 * to Matlab.
 *
 * See template <> mxArray* MxArray::from(const std::vector<glm::tvec2<float>>&)
 * for more details.
 *
 * @param[in] data The data to convert.
 * @return An n x 4 Matlab matrix.
 */
template<>
mxArray* MxArray::from(const std::vector<glm::tvec4<float>>& data)
{
	MxArray out_array(MxArray::Numeric<double>(data.size(), 4));
	for (int r = 0; r < data.size(); ++r) {
		for (int c = 0; c < 4; ++c) {
			out_array.set(r, c, data[r][c]);
		}
	}
	return out_array.release();
};

/**
 * @brief Converts a Matlab matrix into an std::vector of glm::vec4.
 *
 * This function converts an n x 4 Matlab matrix to a vector of vec4's.
 * It is mainly used to pass vertex data from Matlab back to C++.
 *
 * See template <> mxArray* MxArray::from(const std::vector<glm::tvec2<float>>&)
 * for more details regarding the type specialisation.
 *
 * @param[in] in_array The matrix data from Matlab.
 * @param[in,out] data The converted data in C++.
 */
template <>
void MxArray::to(const mxArray* in_array, std::vector<glm::vec4>* data)
{
    MxArray arr(in_array);

    if (arr.dimensionSize() != 2)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 2 dimensions, i.e. n x 4.");
    }
    if (arr.cols() != 4)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 4 elements in the second dimension, i.e. n x 4.");
    }
    if (!arr.isDouble())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array should be of type double.");
    }

    const auto num_vertices = arr.rows();
    data->reserve(num_vertices);

    for (int i = 0; i < num_vertices; ++i)
    {
        data->push_back(
            glm::vec4(arr.at<double>(mwIndex(i), mwIndex(0)), arr.at<double>(mwIndex(i), mwIndex(1)),
                      arr.at<double>(mwIndex(i), mwIndex(2)), arr.at<double>(mwIndex(i), mwIndex(3))));
    }
};

/**
 * @brief Converts an std::vector of std::array<int, 3> to a Matlab matrix.
 *
 * This function converts a whole vector of array<int, 3>'s to an n x 3 Matlab
 * matrix, where n is data.size(). It is mainly used to pass triangle indices
 * data of a Mesh to Matlab.
 *
 * We specialise for vector<array<int, 3>> directly (and not for array<int, 3>)
 * because otherwise a cell array of arrays would be generated. Luckily, even
 * if an array<int, 3> specialisation was to exist too, this one would take
 * precedence to convert a vector<array<int, 3>>.
 *
 * @param[in] data The data to convert.
 * @return An n x 3 Matlab matrix.
 */
template<>
mxArray* MxArray::from(const std::vector<std::array<int, 3>>& data)
{
	MxArray out_array(MxArray::Numeric<int>(data.size(), 3));
	for (int r = 0; r < data.size(); ++r) {
		for (int c = 0; c < 3; ++c) {
			out_array.set(r, c, data[r][c]);
		}
	}
	return out_array.release();
};

/**
 * @brief Converts a Matlab matrix into an std::vector of std::array<int, 3>.
 *
 * This function converts a n x 3 Matlab matrix to a vector<array<int, 3>>.
 * It is mainly used to pass triangle indices data of a Mesh to Matlab.
 *
 * See template <> mxArray* MxArray::from(const std::vector<glm::tvec2<float>>&)
 * for more details regarding the type specialisation.
 *
 * @param[in] in_array The matrix data from Matlab.
 * @param[in,out] data The converted data in C++.
 */
template <>
void MxArray::to(const mxArray* in_array, std::vector<std::array<int, 3>>* data)
{
    MxArray arr(in_array);

    if (arr.dimensionSize() != 2)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 2 dimensions, i.e. n x 3.");
    }
    if (arr.cols() != 3)
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array has to have 3 elements in the second dimension, i.e. n x 3.");
    }
    if (!arr.isInt32())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given array should be of type int32.");
    }

    const auto num_faces = arr.rows();
    data->reserve(num_faces);

    for (int i = 0; i < num_faces; ++i)
    {
        data->push_back(std::array<int, 3>{arr.at<std::int32_t>(mwIndex(i), mwIndex(0)),
                                           arr.at<std::int32_t>(mwIndex(i), mwIndex(1)),
                                           arr.at<std::int32_t>(mwIndex(i), mwIndex(2))});
    }
};

/**
 * @brief Convert an eos::core::Mesh into a Matlab struct.
 *
 * Adjusts the triangle indices from 0-based (C++) to 1-based (Matlab).
 *
 * @param[in] mesh The Mesh that will be returned to Matlab.
 * @return An mxArray containing a Matlab struct with all vertex, colour, texcoords and triangle data.
 */
template<>
mxArray* MxArray::from(const eos::core::Mesh& mesh)
{
	// C++ counts the vertex indices starting at zero, Matlab starts counting
	// at one - therefore, add +1 to all triangle indices:
	auto tvi_1based = mesh.tvi;
	for (auto&& t : tvi_1based) {
		for (auto&& idx : t) {
			idx += 1;
		}
	}
	// Same for tci:
	auto tci_1based = mesh.tci;
	for (auto&& t : tci_1based) {
		for (auto&& idx : t) {
			idx += 1;
		}
	}

	MxArray out_array(MxArray::Struct());
	out_array.set("vertices", mesh.vertices);
	out_array.set("colors", mesh.colors);
	out_array.set("texcoords", mesh.texcoords);
	out_array.set("tvi", tvi_1based);
	out_array.set("tci", tci_1based);

	return out_array.release();
};

/**
 * @brief Convert a Matlab mesh struct back into an eos::core::Mesh.
 *
 * Adjusts the triangle indices from 1-based (Matlab) to 0-based (C++).
 *
 * @param[in] in_array Input mesh data from Matlab.
 * @param[in,out] mesh Converted eos::core::Mesh.
 */
template <>
void MxArray::to(const mxArray* in_array, eos::core::Mesh* mesh)
{
    MxArray array(in_array);

    if (!array.isStruct())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given mesh is not a Matlab struct.");
    }

    if (!array.hasField("vertices") || !array.hasField("tvi"))
    {
        mexErrMsgIdAndTxt("eos:matlab",
                          "Given mesh struct must contain at least the fields 'vertices' and 'tvi'.");
    }

    // We could check whether num_vertices is equal for these, but we'll leave it up to the user to give us
    // valid mesh data.
    array.at("vertices", &mesh->vertices);   // num_vertices x 4 double
    array.at("texcoords", &mesh->texcoords); // num_vertices x 2 double
    array.at("tvi", &mesh->tvi);             // num_faces x 3 int32

    // Adjust the vertex indices from 1-based (Matlab) to 0-based (C++):
    for (auto&& t : mesh->tvi)
    {
        t[0] -= 1;
        t[1] -= 1;
        t[2] -= 1;
    }
};

/**
 * @brief Convert an eos::fitting::Frustum into a Matlab struct.
 *
 * @param[in] frustum The Frustum that will be returned to Matlab.
 * @return An mxArray containing a Matlab struct with all frustum information.
 */
template <>
mxArray* MxArray::from(const eos::fitting::Frustum& frustum)
{
    MxArray out_array(MxArray::Struct());
    out_array.set("l", frustum.l);
    out_array.set("r", frustum.r);
    out_array.set("b", frustum.b);
    out_array.set("t", frustum.t);

    return out_array.release();
};

/**
 * @brief Convert a Matlab frustum struct back into an eos::fitting::Frustum.
 *
 * @param[in] in_array Input mesh data from Matlab.
 * @param[in,out] frustum Converted eos::fitting::Frustum.
 */
template <>
void MxArray::to(const mxArray* in_array, eos::fitting::Frustum* frustum)
{
    MxArray array(in_array);

    if (!array.isStruct())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given frustum is not a Matlab struct.");
    }

    if (!array.hasField("l") || !array.hasField("r") || !array.hasField("b") || !array.hasField("t"))
    {
        mexErrMsgIdAndTxt("eos:matlab",
                          "Given frustum struct must contain the fields 'l', 'r', 'b' and 't'.");
    }

    array.at("l", &frustum->l);
    array.at("r", &frustum->r);
    array.at("b", &frustum->b);
    array.at("t", &frustum->t);
};

/**
 * @brief Define a template specialisation for eos::fitting::RenderingParameters.
 *
 * This converts an eos::fitting::RenderingParameters into a Matlab struct.
 *
 * @param[in] rendering_parameters The RenderingParameters that will be returned to Matlab.
 * @return An mxArray containing a Matlab struct with all required parameters.
 */
template<>
mxArray* MxArray::from(const eos::fitting::RenderingParameters& rendering_parameters) {

	MxArray out_array(MxArray::Struct());

	const std::string camera_type = [&rendering_parameters]() {
		if (rendering_parameters.get_camera_type() == eos::fitting::CameraType::Orthographic)
		{
			return "Orthographic";
		}
		else if (rendering_parameters.get_camera_type() == eos::fitting::CameraType::Perspective) {
			return "Perspective";
		}
		else {
			return "unknown";
		}
	}();

	// Since we don't expose get_opencv_viewport(), and Matlab doesn't have glm::project()
	// anyway, we'll make a 4x4 viewport matrix available. Matlab seems to have the same
	// convention as OpenCV (top-left is the image origin).
	auto viewport = eos::fitting::get_opencv_viewport(rendering_parameters.get_screen_width(), rendering_parameters.get_screen_height());
	glm::mat4x4 viewport_matrix; // Identity matrix
	viewport_matrix[0][0] = 0.5f * viewport[2];
	viewport_matrix[3][0] = 0.5f * viewport[2] + viewport[0];
	viewport_matrix[1][1] = 0.5f * viewport[3];
	viewport_matrix[3][1] = 0.5f * viewport[3] + viewport[1];
	viewport_matrix[2][2] = 0.5f;
	viewport_matrix[3][2] = 0.5f;

	out_array.set("camera_type", camera_type);
	out_array.set("frustum", rendering_parameters.get_frustum());
	out_array.set("rotation_quaternion", rendering_parameters.get_rotation());
	out_array.set("modelview", rendering_parameters.get_modelview());
	out_array.set("projection", rendering_parameters.get_projection());
	out_array.set("viewport", viewport_matrix);
	out_array.set("screen_width", rendering_parameters.get_screen_width());
	out_array.set("screen_height", rendering_parameters.get_screen_height());

	return out_array.release();
};

/**
 * @brief Convert a RenderingParameters Matlab struct back to an eos::fitting::RenderingParameters.
 *
 * @param[in] in_array Input rendering-parameters data from Matlab.
 * @param[in,out] rendering_parameters Converted RenderingParameters.
 */
template <>
void MxArray::to(const mxArray* in_array, eos::fitting::RenderingParameters* rendering_parameters)
{
    MxArray array(in_array);

    if (!array.isStruct())
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given rendering parameters argument is not a Matlab struct.");
    }

    // This is what we're converting for now, mainly to make extract_texture(...) work:
    if (!array.hasField("frustum") || !array.hasField("modelview") || !array.hasField("projection") ||
        !array.hasField("screen_width") || !array.hasField("screen_height"))
    {
        mexErrMsgIdAndTxt("eos:matlab", "Given rendering parameters struct must contain at least the fields "
                                        "'frustum', 'modelview', 'projection', 'screen_width' and 'screen_height'.");
    }

    auto frustum = array.at<eos::fitting::Frustum>("frustum");
    // Get the modelview matrix, extract the rotation quaternion and translation:
    glm::mat4x4 modelview = array.at<glm::mat4x4>("modelview");
    glm::quat rot(modelview);
    auto t_x = modelview[3][0];
    auto t_y = modelview[3][1];

    rendering_parameters->set_rotation(rot);
    rendering_parameters->set_translation(t_x, t_y);
    rendering_parameters->set_screen_width(array.at<int>("screen_width"));
    rendering_parameters->set_screen_height(array.at<int>("screen_height"));
    rendering_parameters->set_frustum(frustum);
};

} /* namespace mexplus */

#endif /* MEXPLUS_EOS_TYPES_HPP_ */


text:
package org.spongepowered.common.event.tracking.phase.packet;

import com.google.common.collect.Lists;
import net.minecraft.entity.item.EntityItem;
import net.minecraft.entity.player.EntityPlayerMP;
import net.minecraft.network.play.client.CPacketClickWindow;
import net.minecraft.network.play.client.CPacketPlayerDigging;
import net.minecraft.util.EnumHand;
import org.spongepowered.api.Sponge;
import org.spongepowered.api.data.Transaction;
import org.spongepowered.api.entity.Entity;
import org.spongepowered.api.event.CauseStackManager;
import org.spongepowered.api.event.CauseStackManager.StackFrame;
import org.spongepowered.api.event.SpongeEventFactory;
import org.spongepowered.api.event.cause.EventContextKeys;
import org.spongepowered.api.event.cause.entity.spawn.SpawnTypes;
import org.spongepowered.api.event.item.inventory.ClickInventoryEvent;
import org.spongepowered.api.item.inventory.Container;
import org.spongepowered.api.item.inventory.ItemStack;
import org.spongepowered.api.item.inventory.ItemStackSnapshot;
import org.spongepowered.api.item.inventory.transaction.SlotTransaction;
import org.spongepowered.common.SpongeImpl;
import org.spongepowered.common.entity.EntityUtil;
import org.spongepowered.common.event.tracking.TrackingUtil;
import org.spongepowered.common.interfaces.IMixinContainer;
import org.spongepowered.common.interfaces.entity.player.IMixinEntityPlayerMP;
import org.spongepowered.common.item.inventory.util.ContainerUtil;
import org.spongepowered.common.item.inventory.util.ItemStackUtil;
import org.spongepowered.common.registry.type.event.InternalSpawnTypes;

import java.util.ArrayList;
import java.util.List;

final class DropItemWithHotkeyState extends BasicInventoryPacketState {

    DropItemWithHotkeyState() {
        super(PacketPhase.MODE_DROP | PacketPhase.BUTTON_PRIMARY | PacketPhase.BUTTON_SECONDARY | PacketPhase.CLICK_ANYWHERE);
    }

    @Override
    public boolean doesCaptureEntityDrops() {
        return true;
    }

    @Override
    public void unwind(InventoryPacketContext context) {
        final EntityPlayerMP player = context.getPacketPlayer();
        //final ItemStack usedStack = context.getItemUsed();
        //final ItemStackSnapshot usedSnapshot = ItemStackUtil.snapshotOf(usedStack);
        final Entity spongePlayer = EntityUtil.fromNative(player);
        try (CauseStackManager.StackFrame frame = Sponge.getCauseStackManager().pushCauseFrame()) {
            Sponge.getCauseStackManager().pushCause(spongePlayer);
            Sponge.getCauseStackManager().addContext(EventContextKeys.SPAWN_TYPE, InternalSpawnTypes.DROPPED_ITEM);

            context.getCapturedBlockSupplier()
                .acceptAndClearIfNotEmpty(blocks -> TrackingUtil.processBlockCaptures(blocks, this, context));
            context.getCapturedItemsSupplier()
                .acceptAndClearIfNotEmpty(items -> {

                    final ArrayList<Entity> entities = new ArrayList<>();
                    for (EntityItem item : items) {
                        entities.add(EntityUtil.fromNative(item));
                    }

                    int usedButton = 0;
                    if (context.getPacket() instanceof CPacketPlayerDigging) {
                        final CPacketPlayerDigging packetIn = context.getPacket();
                        usedButton = packetIn.getAction() == CPacketPlayerDigging.Action.DROP_ITEM ? PacketPhase.PACKET_BUTTON_PRIMARY_ID : 1;
                    } else {
                        final CPacketClickWindow packetIn = context.getPacket();
                        usedButton = packetIn.getUsedButton();
                    }

                    Transaction<ItemStackSnapshot> cursorTrans = new Transaction<>(ItemStackSnapshot.NONE, ItemStackSnapshot.NONE);
                    final IMixinContainer mixinContainer = ContainerUtil.toMixin( player.openContainer);
                    List<SlotTransaction> slotTrans = mixinContainer.getCapturedTransactions();
                    ClickInventoryEvent.Drop dropItemEvent = ((DropItemWithHotkeyState) this)
                        .createInventoryEvent(player, ContainerUtil.fromNative(player.openContainer), cursorTrans, Lists.newArrayList(slotTrans), entities,  usedButton);

                    SpongeImpl.postEvent(dropItemEvent);
                    if (!dropItemEvent.isCancelled() || PacketPhaseUtil.allTransactionsInvalid(dropItemEvent.getTransactions())) {
                        processSpawnedEntities(player, dropItemEvent);
                    } else {
                        ((IMixinEntityPlayerMP) player).restorePacketItem(EnumHand.MAIN_HAND);
                    }
                    slotTrans.clear();
                    mixinContainer.setCaptureInventory(false);
                });
            context.getCapturedEntityDropSupplier()
                .acceptIfNotEmpty(itemMapping -> {

                });
            final IMixinContainer mixinContainer = ContainerUtil.toMixin(player.openContainer);
            mixinContainer.setCaptureInventory(false);
            mixinContainer.getCapturedTransactions().clear();
        }
    }

    @Override
    public ClickInventoryEvent.Drop createInventoryEvent(EntityPlayerMP playerMP, Container openContainer, Transaction<ItemStackSnapshot> transaction,
            List<SlotTransaction> slotTransactions, List<Entity> capturedEntities, int usedButton) {
        try (StackFrame frame = Sponge.getCauseStackManager().pushCauseFrame()) {
            Sponge.getCauseStackManager().addContext(EventContextKeys.SPAWN_TYPE, SpawnTypes.DROPPED_ITEM);
            Sponge.getCauseStackManager().pushCause(openContainer);
            Sponge.getCauseStackManager().pushCause(playerMP);
            for (Entity currentEntity : capturedEntities) {
                currentEntity.setCreator(playerMP.getUniqueID());
            }

            // A 'primary click' is used by the game to indicate a single drop (e.g. pressing 'q' without holding
            // 'control')
            ClickInventoryEvent.Drop event = usedButton == PacketPhase.PACKET_BUTTON_PRIMARY_ID
                    ? SpongeEventFactory.createClickInventoryEventDropSingle(Sponge.getCauseStackManager().getCurrentCause(), transaction,
                            capturedEntities, openContainer, slotTransactions)
                    : SpongeEventFactory.createClickInventoryEventDropFull(Sponge.getCauseStackManager().getCurrentCause(), transaction, capturedEntities,
                            openContainer, slotTransactions);
            // TODO the nature of how this event is handled prevents the cause information being preserved through
            // the event call, somehow should not release this frame until after the event is posted
            return event;
        }
    }

    @Override
    public boolean ignoresItemPreMerges() {
        return true;
    }

}