text:
- Feature Name: Leaseholder Locality ("Leases follow the sun")
- Status: completed
- Start Date: 2017-01-25
- Authors: Alex Robinson
- RFC PR: [#13233](https://github.com/cockroachdb/cockroach/pull/13233)
- Cockroach Issue:
  - [#13232](https://github.com/cockroachdb/cockroach/issues/13232)

# Summary

Enable range leaseholders to move closer to their clients, which reduces the
latency of KV requests when replicas are far apart.

# Motivation

The primary motivation for moving leaseholders around based on where their
requests are coming from is to reduce the latency of those requests.  When a
request for a given key is received by a gateway node, the node must forward the
request to the leaseholder for the key's range. This isn't a big deal if all a
cluster's nodes are nearby each other in the network, but if the leaseholder
happens to be halfway around the world from where the request originated, the
network round trip time that gets added to the request latency by this can be
quite high.

This affects both reads and writes. For reads, getting the request to and from
the leaseholder dominates the request latency in widely distributed clusters
since the leaseholder can serve reads without talking to other replicas. For
writes, even though raft commands will incur an additional round trip between
replicas, removing the round trip to the leaseholder could nearly halve the
total request latency.

While there are typically multiple gateway nodes accessing a range and that they
won't necessarily all be from the same locality, our goal is to minimize the
aggregate RTT cost of accessing a given range by properly placing its
leaseholder.

We believe there are usage patterns that would benefit greatly from better
leaseholder placement. Consider a system that spans datacenters all around the
world. When it's daytime in the Asia/Australia, the datacenter(s) there will be
receiving most of the requests. As time passes, more of the requests will start
to originate from Europe, and later on from the Americas. If the leaseholder for
a range is always in an Asian data center, then the latency of accessing that
range will be much worse when most of its requests come from elsewhere. This is
where the idea of the lease "following the sun" comes from.

Finally, it's worth noting that this goal is somewhat at odds with our desire to
distribute load evenly throughout a cluster (e.g. via [range
rebalancing](rebalancing_v2.md) and [leaseholder
rebalancing](leaseholder_rebalancing.md). In fact, this goal was specifically
pushed off when the initial form of leaseholder rebalancing was designed and
implemented.  Placing the leaseholders near their most common gateways may lower
total throughput if it maxes out what the local machines can do while leaving
the machines in other datacenters underutilized, particularly if the latency
between datacenters is small. There is a fine balance to be kept to avoid
minimizing latency at the expense of too much throughput and vice versa.

# Detailed design

Given that a lease transfer mechanism already exists, the remaining difficulty
lies in deciding when to transfer leases. We [already have
logic](leaseholder_rebalancing.md) that decides when to transfer leases with the
goal of ensuring each node has approximately the same number of leases.
Anything we add to make leases follow the sun will need to place nicely with
that existing goal.

## Tracking request origins

In order to have any idea where to rebalance leases to, we first need to track
where the requests for a given range are coming from. To that end, we can add
request origin information to the `Header` that's included in all
`BatchRequest`s.

When tracking the origin of requests, we don't just care about about the
individual node that a request came from, but its locality. To understand this,
consider a cluster with nodes 1-3 in datacenter 1, nodes 4-6 in datacenter 2,
and nodes 6-9 in datacenter 3. Range x in this cluster has replicas on nodes 1,
4, and 7, and the current leaseholder is node 1. If the leaseholder is receiving
a lot of its requests from nodes 8 and 9, then we may want to move the lease to
node 7 even if node 7 itself isn't sending the range much traffic, because node
7 shares a locality with the origin of the requests.

Luckily, each node already has a `NodeDescriptor` for every other node in the
cluster, and the `NodeDescriptor` proto contains its node's locality
information. Thus, all we need to add to the `BatchRequest` `Header` proto is a
`gateway_node_id` field that gets filled in by the client kv transport. While
the client transport will typically fill this field in with its own ID, it can
also be spoofed when appropriate, such as by DistSQL when a node that wasn't
actually the gateway for a given request makes KV requests on behalf of the real
gateway.

### Alternatives for tracking request origins

Given that we know the IP address of each node in the cluster, we could
potentially try to skip adding the `gateway_node_id` field to each
`BatchRequest` and just rely on the source IP address. That would work in most
cases, but could break down when nodes are communicating with each other via a
load balancer or proxy, without saving much (adding an int to each
`BatchRequest` should have a negligable effect on request size).

We could alternatively include all the locality tags from the source node in
each request, which would eliminate the need to look up the locality of each
node when making decisions based on the collected data. However, this requires
much more data to be sent over the wire and stored per-range in the system,
effectively duplicating the work already done by the gossiping and storage of
node descriptors. It would be reasonable to take this approach -- it would save
an integer's worth of work in the case that the nodes in a cluster don't have
any locality labels -- but it doesn't simplify enough to make up for its added
cost.

## Recording request origins

Each leaseholder replica will maintain a map from locality to the number of
requests received from that locality. Ideally, the request counts would decay
exponentially over time. If this proves too difficult to implement efficiently
(e.g. in a cluster with requests coming from tens of localities), we could opt
for swapping out the map with a new map periodically, such as when we examine it
to decide whether to transfer the replica's lease.

If we go with the latter approach we should also maintain the time that we last
cleared out the map of request counts so that we can determine the rate of
requests to the range. Understanding the load on each range will help us
prioritize them for transfer (and help us to balance the load on each node, even
if that isn't the immediate goal of this work).

For the purposes of optimizing lease locality, we'll count each BatchRequest as
a single request. For tracking the load on each replica, we'll likely want to
measure something a little fancier, such as the number of non-noop subrequests
in the BatchRequest or the amount of KVs scanned by the request. These would be
better estimates of load, but for locality we want to focus on the number of
requests suffering from the large network RTT.

## Making lease transfer decisions

Much of the infrastructure needed for making decisions about when to transfer
leases was [already added](https://github.com/cockroachdb/cockroach/pull/10464)
as part of the initial lease balancing work, so ll we need to do is start using
request locality as an additional input to the decision-making process. The
difficult problems that we really need to solve for are:

* Prioritizing transferring leases for the replicas that are receiving the
  greatest number of requests from distant localities.
* Avoiding thrashing if the origins of requests change frequently.
* Finding the right balance between keeping a similar number of leases on each
  node or moving all the leases to where the most traffic is.

### Prioritizing ranges with the most cross-DC traffic

If a cluster really is spread over datacenters around the world, it's likely
that most of a cluster's ranges will be getting most of their traffic from the
same locality. If this is the case, the most bang-for-the-buck would come from
moving the leases for ranges that are receiving the most requests and whose
request distributions are most skewed toward the given location. To that end, we
may want to periodically record stats about the rate of requests to the ranges
on a store and how many of those requests are from distant localities.  These
would be much like counts we calculate of the number of ranges and leases on a
store. However, unlike those stats, we may not want to start gossiping these as
part of the store descriptor until we have a concrete use for them.  That will
likely come once we start getting smarter about determining just how much load a
node can handle.

Given these stats, we can make decisions per-replica by only transferring leases
for replicas whose requests skew more strongly from distant localities.

Of course, there is a risk that moving all the hottest ranges to the same
locality will could have a worse effect on throughput if those nodes get
overloaded. We'll have to test the heuristics carefully to tune them and
determine when this could become a problem. It's very possible that we'll have
to expose a tuning knob for this to give users more control over the tradeoff.

### Avoiding thrashing

In order to avoid thrashing of leases, we can partially reuse the mechanisms
already in place for avoiding lease thrashing with respect to leaseholder
balance, particularly the rate-limiting of lease transfers
([#11729](https://github.com/cockroachdb/cockroach/pull/11729)).

Additionally, we can learn from both lease and replica rebalancing that there
needs to be a wide range of configurations in which no action is taken -- e.g.
a node with 4% more ranges than the mean won't bother transferring a range to a
node with 4% fewer ranges than the mean. We'll need a cushion in our heuristic
such that we only transfer leases away if there's a significant difference in
the number of requests coming from the different localities.

Along the same lines, we shouldn't make hasty decisions. Measurements of request
distributions are less reliable if we only make them over short time windows.
We'll need to pick a reasonable minimum duration of time to measure before
transferring leases. This will likely be in the tens of minutes -- enough time
to get good data, but not so long that traffic patterns are likely to change
drastically in shorter time periods. An alternative would be to factor in the
number of requests, since a 5 minute measurement with a million samples is much
more trustworthy than a 20 minute measurement with a thousand samples.

### What to do if nodes don't have locality information

If nodes don't have any locality information attached to them, we lose most of
our ability to determine where nodes are located with respect to each other.
While we should make an effort to encourage all multi-datacenter deployments to
specify locality information (and can automate the assignment of locality info
in cloud environments), there will certainly be some deployments without it. We
have a few options in such cases:

* Fall back to operating exactly the same as today, not doing any load-based
  lease placement. We'll still be able to use the per-replica request counts
  once we start factoring load into balancing decisions.
* Fall back to a very limited version of load-based lease placement where we
  only move a lease based on request locality if a node holding one of the other
  replicas is forwarding the vast majority of traffic to the range.
* Calculate (and gossip) estimates of the latency between all pairs of nodes and
  use that to guess at the localities of nodes for the sake of rebalancing.

The last option can be put off to future optimization efforts (if ever). The
second option may be beneficial if a user sets up a multi-datacenter deployment
without locality info, but in the more common case of low-latency, single-DC
deployments it'd likely just make the lease balancing worse for no real latency
gain. We'll stick with the first option for now.

### Reconciling lease locality with lease balance

As mentioned above, our goals in this RFC come into fairly direct conflict with
the goals of the [lease rebalancing RFC](leaseholder_rebalancing.md). If all the
requests are coming from a single locality, it would be ideal from a latency
perspective for all the leases to be there as well, so long as that wouldn't
overload the nodes with too much work. However, it's possible that putting all
the leases into that locality will overload the nodes such that overall cluster
throughput (and even latency) is worse than it would be if the leases were
properly balanced.

Ideally, we would have some understanding of how fully utilized each node is in
terms of throughput. That measurement could be gossiped in the same way as each
node's storage utilization and used in allocation decisions. It's not totally
clear what metric to use for this, though, so for now we will leave it out of
allocation decisions (suggestions welcome -- perhaps a rather naive utilization
measurement can go a long way here). We could easily add a measure of how much
load a node has on it, but not of how close to fully utilized it is.

Similarly, it'd be useful to have an estimate of the RTT between localities when
making allocation decisions. While we don't currently have this (as far as I'm
aware), measuring it wouldn't be very hard, so I suggest that we do so and use
it to help tune lease allocations. We'll have to make sure that this can't lead
to too much thrashing if different nodes come up with different estimates, but
re-measuring this periodically will help fight such issues and also have the
nice benefit that the cluster will react accordingly if the RTT between
localities changes.

Thus, our heuristic for deciding when to transfer a lease will look something
like this:

* If the cluster doesn't have locality information, fall back to the existing
  behavior. The existing behavior is to transfer the lease if the current
  leaseholder node is considered overfull (based on the `rebalanceThreshold`
  constant, which is currently 5% of the mean number of leases) or if it has
  more than the mean number of leases and another replica is underfull.
  * Note that we can start using the new information on how many requests each
    replica has been serving lately to get an idea of how many requests each
    node is serving and use that rather than just the number of leaseholders.
    That work is somewhat orthogonal to optimizing leaseholder locality, but is
    worth noting and working on soon.
* If the cluster does have locality information, then measure the RTT between
  localities. As the latency between localities increases, raise a new
  `interLocalityRebalanceThreshold` proportionately. This will effect the
  underfull/overfull calculations for leases when comparing replicas in
  different localities, but the normal `rebalanceThreshold` would still be used
  when comparing replicas within the same locality. The exact numbers for this
  can be worked out in testing, but it will make lease balancing less and less
  important until we eventually don't consider it at all for replicas in
  different localities. If it's legal to make a cross-locality transfer based on
  the nodes in question and the current `interLocalityRebalanceThreshold`, then
  we can begin considering whether it's worth it to make a transfer based on the
  distributon of requests to the replica.

# Testing

Much like how `allocsim` has been useful for testing the lease rebalancing
heuristics, it would be very nice to have a repeatable tool for testing this as
well. However, whereas the only real variable when testing lease rebalancing was
the relevant code and the only real output was the number of leases held by each
node, we now have multiple inputs (the code, the latency between nodes, the
locality labels) and multiple outputs (the number of leases held by each node,
the distribution of request latency, and the request throughput) to consider.

In order to better test this new functionality, I propose:

* Adding a testing knob to simulate additional latency between nodes.
* Extending `allocsim` to be able to set up different locality configurations
  with the new latency knob.
* Extending `allocsim` to send load against the specific nodes/localities and
  measure the resulting throughput and latency.

The output of allocsim will enable us to better understand how different
heuristics perform on different cluster configurations. We may be able to find a
heuristic that performs reasonably well across a variety of clusters, or may
find that we have to introduce a tuning knob of some sort to shift the balance
one way or another. Either way, it's tough for us to know exactly what'll work
without a testing tool like this, so we'll rely heavily on it when tuning the
heuristic.

# Future Directions

* It would be helpful to expose information about balancing decisions in the UI
  so that users and developers can better understand what's happening in their
  clusters, which will be particularly important when bad decisions are made,
  causing performance dips.

* There could be situations in which the locality that's generating most of the
  requests to a range doesn't have a local replica for that range. While ideally
  cluster admins would construct `ZoneConfig` settings that make sense for their
  environments and workloads, we could potentially benefit from taking
  per-locality load into account when making replica placement decisions (not
  just lease placement decisions).

* As mentioned above, we should start using the recent request counts being
  added as part of this work to improve the existing replica and lease placement
  heuristics. This should be fairly straightforward, it'll just require some
  benchmarking. There's more discussion of this in the [future directions
  section of the leaseholder rebalancing
  RFC](leaseholder_rebalancing.md#future-directions).

* As mentioned above, it would be very beneficial if we came up with some way of
  measuring the true load on each node, either as a collection of measurements
  or as some combination of CPU utilization, memory pressure, and disk I/O
  utilization. This would make improving locality and balancing load
  significantly easier and more effective. Ideally we could even differentiate
  between load caused by being a leaseholder and load caused by being a follower
  replica.

* It's possible that this work will actually be harmful in certain odd cluster
  configurations. For example, if a range has two replicas in different parts of
  Europe and only a single replica in Australia, then writes will perform better
  if the lease is in Europe even if most of its requests are coming from
  Australia. This is because raft commands will commit much more quickly if
  proposed from one of the replicas in Europe. We may want to take into account
  the latency between the localities of all the different replicas to avoid such
  problems, but such cases aren't critical since they're not recommended
  configurations to begin with.

# Drawbacks

* This approach does not have any true safeguards against overloading nodes in
  localities where most requests are coming from. We will prefer moving load
  there as long as the inter-locality RTT is high enough regardless of how much
  load the nodes can handle. At a high level, this is an existing issue of our
  system, but intentionally imbalancing the leases will make the risk much worse
  (until we start factoring in some measure of load percentage when making
  decisions). We're going to need some sort of flow control mechanism regardless
  of this change, and once we have it we can use it to help with these
  decisions.
* Relying on measurements of latency from each node to each other locality may
  lead to unexpected thrashing if nodes get drastically different measurements.
  Taking multiple measurements over time should help with this, but it's
  conceivable that certain networks could exhibit a persistent difference.
* Unless we decide to do a lot more work determining and gossiping latency
  information between all nodes, these optimizations won't be used if cluster
  admins don't add locality info to the nodes.

# Alternatives

* As mentioned above, we could track the origin of each request by sending the
  locality labels along with each request rather than just the source node ID,
  but that would add a lot of duplicated info along with every request.
* Rather than considering it future work, it'd be beneficial if we could make
  allocation decisions based on the actual load on each node. Skipping right to
  that solution would be great (suggestions appreciated!).
* We may find it more effective to measure something other than number of
  requests handled by a replica. For example, perhaps time spent processing
  requeusts for the replica or bytes returned in response to requests to the
  replica would be more accurate measurements of the load on the replica. These
  are things that we could potentially experiment with using allocsim to see if
  they provide better balance.

# Unresolved questions


text:
##

 Hjson Syntax

TL;DR [here](/)

%%%

### Details

#### Commas

You can separate your values/members either with a comma or a new line.

Unlike JSON, trailing commas are ignored and do not produce a syntax error.

For example:

```
{
  one: 1
  two: 2,
  more: [3,4,5]
  trailing: 6,
}
```

You should omit optional commas to make your data more readable.

```
{
  one: 1
  two: 2
}
```

#### Whitespace & Comments

Whitespace and comments can be inserted between any pair of tokens.

Hash `#` or C style `//` comments start a single line comment.

The style is just a matter of preference. `#` may be easier on the eyes but `//` is sometimes used in existing JSON configs.

`/*` starts a multiline comment that ends with `*/`. You can use it to comment out part of the data.

#### Keys

You only need to add quotes if the key name includes whitespace or any of the punctuators (`{}[],:`).

For example:

- `foo`
- `"test case"`
- `"{option}"`

#### Quoteless Strings

A quoteless string cannot start with any of the punctuators (`{}[],:`).

Unlike a quoted string it automatically ends at the end of the current line. Do not add commas or comments as they would become part of the string. If you wish to add comments place them on the previous or next line.

Preceding and trailing whitespace is ignored. Escapes are only supported in quoted strings.

The Hjson parser will still detect values (*number*, `true`, `false` or `null`) and parse them correctly. For example

- `3` is the *number* `3`
- `5 times` is the *string* `"5 times"`
- `true` is the *boolean* `true`
- `7 # minutes` is the *number* `7` followed by a comment
- `\s#([0-9a-fA-F]{3})` is the *string* `"\\s#([0-9a-fA-F]{3})"`

#### Multiline Strings

- Start with triple quotes `'''`, whitespace on the first line is ignored
- `'''` defines the head, on the following lines all whitespace up to this column is ignored
- all other whitespace is assumed to be part of the string.
- ends with triple quotes `'''`. The last newline is ignored to allow for better formatting.

A multiline string is OS and file independent. The line feed is always `\n`.

For example

```
haiku:
  '''
  My half empty glass,
  I will fill your empty half.
  Now you are half full.
  '''
```

Is the *string* `"My half empty glass,\nI will fill your empty half.\nNow you are half full."`

#### Strings

Quoted strings can be written with 'single' or "double" quotes.

Note: single quotes are new (2017/7) and may not yet be supported on all platforms.

#### Root Braces

Hjson allows you to omit `{}` for the root object so that a config file can contain nothing but keys and values, similar to YAML.

Note: This feature was previously marked as deprecated but is now officially supported again. If you need maximum portability with your Hjson files, then we recommend you keep the root braces in case a third party implementation doesn't support it.

#### File extension

`.hjson`

#### Encoding

`.hjson` files must be encoded in UTF-8 (without a BOM).

#### Header

Hjson does not have a header but if you want to indicate the file type you can use `#hjson` on the first line.

#### RFC

View the RFC draft (pending updates/root braces/single quotes)

redpajama_set_name: RedPajamaGithub


text:
package opennlp.scalabha.tag.hmm

import scala.Array.canBuildFrom
import scala.io.Source
import org.apache.commons.logging.LogFactory
import org.apache.log4j.Level
import org.apache.log4j.Logger
import org.junit.Assert._
import org.junit.BeforeClass
import org.junit.Test
import opennlp.scalabha.tag.hmm._
import opennlp.scalabha.tag.hmm.support._
import opennlp.scalabha.tag.support._
import opennlp.scalabha.tag._
import opennlp.scalabha.util.CollectionUtil._
import opennlp.scalabha.util.CollectionUtils._
import opennlp.scalabha.util.FileUtils

class UnsupervisedEmHmmTaggerTrainerTests {
  val LOG = LogFactory.getLog(classOf[UnsupervisedEmHmmTaggerTrainerTests])

  @Test
  def tiny_beforeEM() {
    val trainRaw = Vector(
      "the dog walks quickly",
      "the cat walks quietly",
      "the dog saw the cat",
      "the cat saw the dog",
      "the dog saw the saw",
      "the bird sings",
      "the mouse walks",
      "the aardvark walks",
      "the aardvark meanders").map(_.split(" ").toVector)

    val tagDict = SimpleTagDict(Map(
      "bird" -> Set("N"),
      "cat" -> Set("N"),
      "dog" -> Set("N"),
      "horse" -> Set("N"),
      "mouse" -> Set("N"),
      "quickly" -> Set("R"),
      "quietly" -> Set("R"),
      "saw" -> Set("N", "V"),
      "sings" -> Set("V"),
      "the" -> Set("D"),
      "walks" -> Set("V")))

    val gold = Vector(
      Vector(("the", "D"), ("bird", "N"), ("walks", "V")),
      Vector(("the", "D"), ("horse", "N"), ("walks", "V")),
      Vector(("the", "D"), ("aardvark", "N"), ("walks", "V")),
      Vector(("the", "D"), ("dog", "N"), ("meanders", "V")),
      Vector(("a", "D"), ("dog", "N"), ("walks", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("moose", "N"), ("walks", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("dog", "N"), ("runs", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("dog", "N"), ("walks", "V"), ("briskly", "R")))

    // Create the initial distributions
    val allTags = tagDict.allTags.map(Option(_)) + None
    val initialTransitions = CondFreqDist(DefaultedCondFreqCounts.fromMap(allTags.mapToVal(allTags.mapToVal(1.).toMap).toMap))
    val initialEmissions =
      new EstimatedRawCountUnsupervisedEmissionDistFactory(
        new PassthroughCountsTransformer(),
        tagDict,
        trainRaw).make()
    val unsupervisedTagger = HmmTagger(initialTransitions, initialEmissions, tagDict.opt)

    val output = unsupervisedTagger.tag(gold.map(_.map(_._1)))
    val results = new TaggerEvaluator().evaluate(output, gold, tagDict)
    assertResultsEqual("""
		Total:   53.57 (15/28)
		Known:   59.09 (13/22)
		Unknown: 33.33 (2/6)
		Common Mistakes:
		#Err     Gold      Model
		6        V        N
		4        R        N
		3        D        N
    	""", results)
  }

  @Test
  def tiny() {
    val trainRaw = Vector(
      "the aardvark walks",
      "the aardvark meanders",
      "the dog walks quickly",
      "the cat walks quietly",
      "the dog saw the cat",
      "the cat saw the dog",
      "the dog saw the saw",
      "the bird sings",
      "the mouse walks").map(_.split(" ").toVector)

    val tagDict = SimpleTagDict(Map(
      "bird" -> Set("N"),
      "cat" -> Set("N"),
      "dog" -> Set("N"),
      "horse" -> Set("N"),
      "mouse" -> Set("N"),
      "quickly" -> Set("R"),
      "quietly" -> Set("R"),
      "saw" -> Set("N", "V"),
      "sings" -> Set("V"),
      "the" -> Set("D"),
      "walks" -> Set("V")))

    val gold = Vector(
      Vector(("a", "D"), ("dog", "N"), ("walks", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("bird", "N"), ("walks", "V")),
      Vector(("the", "D"), ("horse", "N"), ("walks", "V")),
      Vector(("the", "D"), ("aardvark", "N"), ("walks", "V")),
      Vector(("the", "D"), ("moose", "N"), ("walks", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("dog", "N"), ("runs", "V"), ("quickly", "R")),
      Vector(("the", "D"), ("dog", "N"), ("walks", "V"), ("briskly", "R")))

    val initialHmm = HmmTagger(
      HmmUtils.uniformTransitionDist(tagDict.allTags),
      new EstimatedRawCountUnsupervisedEmissionDistFactory(
        new PassthroughCountsTransformer(),
        tagDict,
        trainRaw).make(),
      tagDict.opt)

    val unsupervisedTrainer: TypesupervisedTaggerTrainer[String, String] =
      new EmHmmTaggerTrainer(
        transitionCountsTransformer =
          new TransitionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1.)),
        emissionCountsTransformer =
          new EmissionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1., AddLambdaSmoothingCountsTransformer(1.))),
        hmmTaggerFactory = new HardTagDictConstraintHmmTaggerFactory(OptionalTagDict(tagDict)),
        maxIterations = 1,
        minAvgLogProbChangeForEM = 0.00001)
    val unsupervisedTagger = unsupervisedTrainer.train(trainRaw, tagDict)
    val output = unsupervisedTagger.tag(gold.map(_.map(_._1)))
    val results = new TaggerEvaluator().evaluate(output, gold, tagDict)
    assertResultsEqual("""
		Total:   100.00 (25/25)
		Known:   100.00 (20/20)
		Unknown: 100.00 (5/5)
		Common Mistakes:
		#Err     Gold      Model
    	""", results)
  }

  @Test
  def ic_fullTagDict_noSmoothing() {
    val trainLab = TaggedFile("data/postag/ic/ictrain.txt")
    val testLab = TaggedFile("data/postag/ic/ictest.txt")

    val testRaw = AsRawFile("data/postag/ic/ictest.txt")
    val gold = TaggedFile("data/postag/ic/ictest.txt")

    val tagDict = new SimpleTagDictFactory().make(trainLab ++ testLab)

    val initialHmm = HmmTagger(
      HmmUtils.uniformTransitionDist(tagDict.allTags),
      new EstimatedRawCountUnsupervisedEmissionDistFactory(
        new PassthroughCountsTransformer(),
        tagDict,
        trainLab.map(_.map(_._1))).make(),
      tagDict.opt)

    val unsupervisedTrainer: TypesupervisedTaggerTrainer[String, String] =
      new EmHmmTaggerTrainer(
        transitionCountsTransformer =
          new TransitionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1.)),
        emissionCountsTransformer =
          new EmissionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1., AddLambdaSmoothingCountsTransformer(1.))),
        hmmTaggerFactory = new HardTagDictConstraintHmmTaggerFactory(OptionalTagDict(tagDict)),
        maxIterations = 20,
        minAvgLogProbChangeForEM = 0.00001)
    val unsupervisedTagger = unsupervisedTrainer.train(trainLab.map(_.map(_._1)), tagDict)
    val output = unsupervisedTagger.tag(testRaw)
    val results = new TaggerEvaluator().evaluate(output, gold, tagDict)
    assertResultsEqual("""
		Total:   48.48 (16/33)
		Known:   48.48 (16/33)
		Unknown: NaN (0/0)
		Common Mistakes:
		#Err     Gold      Model
		17       H        C
    	""", results)
  }

  @Test
  def en_largeTagDict() {
    val trainLab = TaggedFile("data/postag/english/entrain")
    val tagDict = new SimpleTagDictFactory().make(trainLab)
    val unsupervisedResults = runUnsupervisedTrainingTest(tagDict)
    assertResultsEqual("""
		Total:   89.16 (21353/23949)
		Known:   92.48 (20198/21841)
		Unknown: 54.79 (1155/2108)
		Common Mistakes:
		#Err     Gold      Model
		522      D        F
		375      N        J
		223      N        V
		208      V        N
		122      J        N
    	""", unsupervisedResults)
  }

  @Test
  def en_smallTagDict() {
    val tagDictTrain = TaggedFile("data/postag/english/entrain").take(3000)
    val tagDict = new SimpleTagDictFactory().make(tagDictTrain)
    val unsupervisedResults = runUnsupervisedTrainingTest(tagDict)
    assertResultsEqual("""
		Total:   89.98 (21549/23949)
		Known:   94.56 (20301/21469)
		Unknown: 50.32 (1248/2480)
		Common Mistakes:
		#Err     Gold      Model
		458      N        J
		230      V        N
		224      N        V
		132      N        D
		126      J        N
    	""", unsupervisedResults)
  }

  private def runUnsupervisedTrainingTest(tagDict: TagDict[String, String]) = {
    val trainRaw = RawFile("data/postag/english/enraw20k")
    val gold = TaggedFile("data/postag/english/entest")

    LOG.debug("tagDictTrain.size = " + tagDict.setIterator.ungroup.size)
    LOG.debug("labeledTrain.size = " + 0)
    LOG.debug("rawTrain.size     = " + trainRaw.size)

    val initialHmm = HmmTagger(
      HmmUtils.uniformTransitionDist(tagDict.allTags),
      new EstimatedRawCountUnsupervisedEmissionDistFactory(
        new PassthroughCountsTransformer(),
        tagDict,
        trainRaw).make(),
      tagDict.opt)

    val unsupervisedTrainer: TypesupervisedHmmTaggerTrainer[String,String] =
      new EmHmmTaggerTrainer[String, String](
        transitionCountsTransformer =
          new TransitionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1.)),
        emissionCountsTransformer =
          new EmissionCountsTransformer(
            EisnerSmoothingCondCountsTransformer(1., AddLambdaSmoothingCountsTransformer(1.))),
        hmmTaggerFactory = new HardTagDictConstraintHmmTaggerFactory(OptionalTagDict(tagDict)),
        maxIterations = 20,
        minAvgLogProbChangeForEM = 0.00001)

    val unsupervisedTagger = unsupervisedTrainer.trainFromInitialHmm(trainRaw, initialHmm, tagDict)
    val unsupervisedOutput = unsupervisedTagger.tag(gold.map(_.map(_._1)))
    new TaggerEvaluator().evaluate(unsupervisedOutput, gold, tagDict)
  }

  object TaggedFile {
    def apply(filename: String) =
      FileUtils.readLines(filename)
        .map(_.trim)
        .split("###/###")
        .filter(_.nonEmpty)
        .map(_.map(_.split("/").toTuple2).toVector)
        .toVector
  }

  object AsRawFile {
    def apply(filename: String) =
      TaggedFile(filename).map(_.map(_._1))
  }

  object RawFile {
    def apply(filename: String) =
      FileUtils.readLines(filename)
        .map(_.trim)
        .split("###")
        .filter(_.nonEmpty)
        .map(_.toVector)
        .toVector
  }

  def assertResultsEqual(expectedString: String, results: ScoreResults[String, String]) {
    assertEquals(
      expectedString.split("\n").map(_.trim).filter(_.nonEmpty).mkString("\n"),
      results.toString.split("\n").map(_.trim).filter(_.nonEmpty).mkString("\n"))
  }
}

object UnsupervisedEmHmmTaggerTrainerTests {

  @BeforeClass def turnOffLogging() {
    Logger.getRootLogger.setLevel(Level.OFF)
  }

}


text:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.7"/>
<title>Occluded Library: occluded::opengl::retained::shaders::shader Class Reference</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>

<link href="navtree.css" rel="stylesheet" type="text/css"/>

<link href="search/search.css" rel="stylesheet" type="text/css"/>

<link href="doxygen.css" rel="stylesheet" type="text/css" />

Occluded Library
    0.1

A graphics library that sits on top of low level libraries such as OpenGL and DirectX.

Main Page

Classes

Files

          <input type="text" id="MSearchField" value="Search" accesskey="S"
               onfocus="searchBox.OnSearchFieldFocus(true)"
               onblur="searchBox.OnSearchFieldFocus(false)"
               onkeyup="searchBox.OnSearchFieldChange(event)"/>

Class List

Class Index

Class Hierarchy

Class Members

 All Classes Functions

Public Member Functions |
List of all members

occluded::opengl::retained::shaders::shader Class Reference

A wrapper class for an OpenGL shader.
 More...

#include <shader.h>

Public Member Functions

 	shader (const std::string &shaderSrc, const shader_type_t type)

 	Initializes and compiles the shader.  More...

const GLuint 	get_id () const

 	Gets the id of the shader.  More...

const bool 	is_compiled () const

 	Returns true if the shader is compiled, and false otherwise.  More...

const shader_type_t 	get_type () const

 	Gets the type of the shader.  More...

const std::string & 	get_compile_log () const

 	Gets the compile log if a compilation error occurs.  More...

Detailed Description

A wrapper class for an OpenGL shader.

A wrapper class for an OpenGL GLSL shader. This is class holds information necessary for creating a GLSL shader as well as making it easy for accessing information about a compiled shader without having to query OpenGL. It also allows for determining if a shader has not compiled properly and for accessing the error log to examine the compilation errors.

Constructor & Destructor Documentation

          occluded::opengl::retained::shaders::shader::shader
          (
          const std::string &
          shaderSrc,

          const shader_type_t
          type

          )

Initializes and compiles the shader.

Parameters

shaderSrc	A reference to a string containing the source code of the GLSL shader.

type	The type of this GLSL shader.

Initializes and compiles the shader program specified by the shader source and the shader type provided. If the shader has a problem compiling, the compile log will be stored and an exception will be thrown.

Member Function Documentation

          occluded::opengl::retained::shaders::shader::get_compile_log
          (
          	)
           const

Gets the compile log if a compilation error occurs.

Returns

Returns a reference to the string containing the shader's compile log.

Gets the the compile log of the GLSL shader if there is a problem during compilation. If there was no problem during compilation, an exception will be thrown.

          occluded::opengl::retained::shaders::shader::get_id
          (
          	)
           const

Gets the id of the shader.

Returns

Returns the id of the GLSL shader.

Gets the id of the GLSL shader and throws an exception if an error occured during shader compilation.

          occluded::opengl::retained::shaders::shader::get_type
          (
          	)
           const

Gets the type of the shader.

Returns

Returns a shader_type_t representing the shader's type.

Gets the shader_type_t of the shader. Used to prevent problems with incorrect types of shaders being used in the same shader_program.

          occluded::opengl::retained::shaders::shader::is_compiled
          (
          	)
           const

Returns true if the shader is compiled, and false otherwise.

Returns

Returns whether or not the shader is compiled.

Returns true if the GLSL is compiled and false if an error occured during compilation. This is used to quickly determine if a shader can be used in a GLSL shader program.

The documentation for this class was generated from the following files:

OccludedLibrary/opengl/retained/shaders/shader.h

OccludedLibrary/opengl/retained/shaders/shader.cpp

occluded

opengl

retained

shaders

shader

Generated on Mon Aug 25 2014 22:44:31 for Occluded Library by

     1.8.7


text:
#ifndef PG_SHMEM_H
#define PG_SHMEM_H

typedef struct PGShmemHeader	/* standard header for all Postgres shmem */
{
	int32		magic;			/* magic # to identify Postgres segments */
#define PGShmemMagic  679834894
	pid_t		creatorPID;		/* PID of creating process */
	Size		totalsize;		/* total size of segment */
	Size		freeoffset;		/* offset to first free space */
	Size		indexoffset;	/* offset to ShmemIndex table */
#ifndef WIN32					/* Windows doesn't have useful inode#s */
	dev_t		device;			/* device data directory is on */
	ino_t		inode;			/* inode number of data directory */
#endif
} PGShmemHeader;

#ifdef EXEC_BACKEND
extern unsigned long UsedShmemSegID;
extern void *UsedShmemSegAddr;

extern void PGSharedMemoryReAttach(void);
#endif

extern PGShmemHeader *PGSharedMemoryCreate(Size size, bool makePrivate,
					 int port);
extern bool PGSharedMemoryIsInUse(unsigned long id1, unsigned long id2);
extern void PGSharedMemoryDetach(void);

#endif   /* PG_SHMEM_H */


text:
var path = require('path');
var htmlWebpackPlugin = require('html-webpack-plugin');

module.exports = {
	entry:path.resolve(__dirname, './src/index.js'),
	output:{
		path:path.resolve(__dirname, 'dist'),
		filename:'bundle.js'
	},
	module:{
		loaders:[
			{
				test:/\.js$/,
				loader:'babel-loader',
				exclude:/node_modules/
			},
			{
				test:/\.css/,
				loader:'style-loader!css-loader'
			}
		]
	},
	plugins:[
		new htmlWebpackPlugin({
			title:'React第一个组件',
			template:path.resolve(__dirname, './src/index.html')
		})
	]
}


text:
struct quickscope_wrapper {
	std::function<void()> destructor;

	~quickscope_wrapper();
};


text:
<!DOCTYPE html>

        <title>DataChannel.js » A WebRTC Library for Data Sharing</title>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">
        <link rel="author" type="text/html" href="https://plus.google.com/+MuazKhan">
        <meta name="author" content="Muaz Khan">
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">

        <link rel="stylesheet" href="https://cdn.webrtc-experiment.com/style.css">

                    DataChannel.js
                    » A WebRTC Library for Data Sharing

                    HOME
                     ©
                    Muaz Khan

                    .
                    @WebRTCWeb

                    .
                    Github

                    .
                    Latest issues

                    .
                    What's New?

Open New DataChannel Connection

                    <button id="open-channel">Open</button>

Text Chat

                            <input type="text" id="chat-input" style="font-size: 1.2em;" placeholder="chat message" disabled>

Share Files

                            <input type="file" id="file" disabled>

DataChannel.js Features:

Direct messages — to any user using his `user-id`

Eject/Reject any user — using his `user-id`

Leave any room (i.e. data session) or close entire session using `leave` method

File size is limitless!

Text message length is limitless!

Size of data is also limitless!

Fallback to socket.io/websockets/etc.

Users' presence detection using `onleave`

Latency detection

Multi-longest strings/files concurrent transmission

How to use DataChannel.js?

<script src="https://cdn.webrtc-experiment.com/DataChannel.js"> </script>

<input type="text" id="chat-input" disabled
       style="font-size: 2em; width: 98%;"><br />
<div id="chat-output"></div>

<script>
    var chatOutput = document.getElementById('chat-output');
    var chatInput = document.getElementById('chat-input');
    chatInput.onkeypress = function(e) {
        if (e.keyCode != 13) return;
        channel.send(this.value);
        chatOutput.innerHTML = 'Me: ' + this.value + '<hr />'
                             + chatOutput.innerHTML;
        this.value = '';
    };
</script>

<script>
    var channel = new DataChannel('Session Unique Identifier');

    channel.onopen = function(userid) {
        chatInput.disabled = false;
        chatInput.value = 'Hi, ' + userid;
        chatInput.focus();
    };

    channel.onmessage = function(message, userid) {
        chatOutput.innerHTML = userid + ': ' + message + '<hr />'
                             + chatOutput.innerHTML;
    };

    channel.onleave = function(userid) {
        chatOutput.innerHTML = userid + ' Left.<hr />'
                             + chatOutput.innerHTML;
    };
</script>

Use your own socket.io for signaling

<script>
    // by default socket.io is used for signaling; you can override it
    channel.openSignalingChannel = function(config) {
        var socket = io.connect('http://your-site:8888');
        socket.channel = config.channel || this.channel || 'default-channel';
        socket.on('message', config.onmessage);

        socket.send = function (data) {
            socket.emit('message', data);
        };

        if (config.onopen) setTimeout(config.onopen, 1);
        return socket;
    }
</script>

Latest Updates

Feedback

                    <textarea id="message" style="border: 1px solid rgb(189, 189, 189); height: 8em; margin: .2em; outline: none; resize: vertical; width: 98%;" placeholder="Have any message? Suggestions or something went wrong?"></textarea>

                <button id="send-message" style="font-size: 1em;">Send Message</button>Enter your email too; if you want "direct" reply!

                WebRTC Experiments
                © Muaz Khan
                muazkh@gmail.com


text:
<?php

namespace ApiBird\Extension;

class Form extends \Phalcon\DI\Injectable implements \ApiBird\ExtensionInterface
{

    /**
     * Mime types parsed
     * @var array
     */
    protected static $types = [
        'application/x-www-form-urlencoded',
    ];

    /**
     * Parse type from format
     * @param string $data
     * @return array
     */
    public function fromFormat($data)
    {
        $output = null;
        parse_str($data, $output);
        return $output;
    }

    /**
     * Parse type to format
     * @param array $data
     * @return string
     */
    public function toFormat($data)
    {
        return http_build_query($data);
    }

    public static function getTypes()
    {
        return static::$types;
    }

}


text:
using System.Collections.Generic;
using System.Diagnostics;

namespace System.Net.Http.Headers
{
    // The purpose of this type is to extract the handling of general headers in one place rather than duplicating
    // functionality in both HttpRequestHeaders and HttpResponseHeaders.
    internal sealed class HttpGeneralHeaders
    {
        private HttpHeaderValueCollection<string> _connection;
        private HttpHeaderValueCollection<string> _trailer;
        private HttpHeaderValueCollection<TransferCodingHeaderValue> _transferEncoding;
        private HttpHeaderValueCollection<ProductHeaderValue> _upgrade;
        private HttpHeaderValueCollection<ViaHeaderValue> _via;
        private HttpHeaderValueCollection<WarningHeaderValue> _warning;
        private HttpHeaderValueCollection<NameValueHeaderValue> _pragma;
        private HttpHeaders _parent;
        private bool _transferEncodingChunkedSet;
        private bool _connectionCloseSet;

        public CacheControlHeaderValue CacheControl
        {
            get { return (CacheControlHeaderValue)_parent.GetParsedValues(HttpKnownHeaderNames.CacheControl); }
            set { _parent.SetOrRemoveParsedValue(HttpKnownHeaderNames.CacheControl, value); }
        }

        public HttpHeaderValueCollection<string> Connection
        {
            get { return ConnectionCore; }
        }

        public bool? ConnectionClose
        {
            get
            {
                // If we've already initialized the connection header value collection
                // and it contains the special value, or if we haven't and the headers contain
                // the parsed special value, return true.  We don't just access ConnectionCore,
                // as doing so will unnecessarily initialize the collection even if it's not needed.
                if (_connection != null)
                {
                    if (_connection.IsSpecialValueSet)
                    {
                        return true;
                    }
                }
                else if (_parent.ContainsParsedValue(HttpKnownHeaderNames.Connection, HeaderUtilities.ConnectionClose))
                {
                    return true;
                }
                if (_connectionCloseSet)
                {
                    return false;
                }
                return null;
            }
            set
            {
                if (value == true)
                {
                    _connectionCloseSet = true;
                    ConnectionCore.SetSpecialValue();
                }
                else
                {
                    _connectionCloseSet = value != null;
                    ConnectionCore.RemoveSpecialValue();
                }
            }
        }

        public DateTimeOffset? Date
        {
            get { return HeaderUtilities.GetDateTimeOffsetValue(HttpKnownHeaderNames.Date, _parent); }
            set { _parent.SetOrRemoveParsedValue(HttpKnownHeaderNames.Date, value); }
        }

        public HttpHeaderValueCollection<NameValueHeaderValue> Pragma
        {
            get
            {
                if (_pragma == null)
                {
                    _pragma = new HttpHeaderValueCollection<NameValueHeaderValue>(HttpKnownHeaderNames.Pragma, _parent);
                }
                return _pragma;
            }
        }

        public HttpHeaderValueCollection<string> Trailer
        {
            get
            {
                if (_trailer == null)
                {
                    _trailer = new HttpHeaderValueCollection<string>(HttpKnownHeaderNames.Trailer,
                        _parent, HeaderUtilities.TokenValidator);
                }
                return _trailer;
            }
        }

        public HttpHeaderValueCollection<TransferCodingHeaderValue> TransferEncoding
        {
            get { return TransferEncodingCore; }
        }

        internal static bool? GetTransferEncodingChunked(HttpHeaders parent, HttpGeneralHeaders headers)
        {
            // If we've already initialized the transfer encoding header value collection
            // and it contains the special value, or if we haven't and the headers contain
            // the parsed special value, return true.  We don't just access TransferEncodingCore,
            // as doing so will unnecessarily initialize the collection even if it's not needed.
            if (headers != null && headers._transferEncoding != null)
            {
                if (headers._transferEncoding.IsSpecialValueSet)
                {
                    return true;
                }
            }
            else if (parent.ContainsParsedValue(HttpKnownHeaderNames.TransferEncoding, HeaderUtilities.TransferEncodingChunked))
            {
                return true;
            }
            if (headers != null && headers._transferEncodingChunkedSet)
            {
                return false;
            }
            return null;
        }

        public bool? TransferEncodingChunked
        {
            get
            {
                // Separated out into a static to enable access to TransferEncodingChunked
                // without the caller needing to force the creation of HttpGeneralHeaders
                // if it wasn't created for other reasons.
                return GetTransferEncodingChunked(_parent, this);
            }
            set
            {
                if (value == true)
                {
                    _transferEncodingChunkedSet = true;
                    TransferEncodingCore.SetSpecialValue();
                }
                else
                {
                    _transferEncodingChunkedSet = value != null;
                    TransferEncodingCore.RemoveSpecialValue();
                }
            }
        }

        public HttpHeaderValueCollection<ProductHeaderValue> Upgrade
        {
            get
            {
                if (_upgrade == null)
                {
                    _upgrade = new HttpHeaderValueCollection<ProductHeaderValue>(HttpKnownHeaderNames.Upgrade, _parent);
                }
                return _upgrade;
            }
        }

        public HttpHeaderValueCollection<ViaHeaderValue> Via
        {
            get
            {
                if (_via == null)
                {
                    _via = new HttpHeaderValueCollection<ViaHeaderValue>(HttpKnownHeaderNames.Via, _parent);
                }
                return _via;
            }
        }

        public HttpHeaderValueCollection<WarningHeaderValue> Warning
        {
            get
            {
                if (_warning == null)
                {
                    _warning = new HttpHeaderValueCollection<WarningHeaderValue>(HttpKnownHeaderNames.Warning, _parent);
                }
                return _warning;
            }
        }

        private HttpHeaderValueCollection<string> ConnectionCore
        {
            get
            {
                if (_connection == null)
                {
                    _connection = new HttpHeaderValueCollection<string>(HttpKnownHeaderNames.Connection,
                        _parent, HeaderUtilities.ConnectionClose, HeaderUtilities.TokenValidator);
                }
                return _connection;
            }
        }

        private HttpHeaderValueCollection<TransferCodingHeaderValue> TransferEncodingCore
        {
            get
            {
                if (_transferEncoding == null)
                {
                    _transferEncoding = new HttpHeaderValueCollection<TransferCodingHeaderValue>(
                        HttpKnownHeaderNames.TransferEncoding, _parent, HeaderUtilities.TransferEncodingChunked);
                }
                return _transferEncoding;
            }
        }

        internal HttpGeneralHeaders(HttpHeaders parent)
        {
            Debug.Assert(parent != null);

            _parent = parent;
        }

        internal static void AddParsers(Dictionary<string, HttpHeaderParser> parserStore)
        {
            Debug.Assert(parserStore != null);

            parserStore.Add(HttpKnownHeaderNames.CacheControl, CacheControlHeaderParser.Parser);
            parserStore.Add(HttpKnownHeaderNames.Connection, GenericHeaderParser.TokenListParser);
            parserStore.Add(HttpKnownHeaderNames.Date, DateHeaderParser.Parser);
            parserStore.Add(HttpKnownHeaderNames.Pragma, GenericHeaderParser.MultipleValueNameValueParser);
            parserStore.Add(HttpKnownHeaderNames.Trailer, GenericHeaderParser.TokenListParser);
            parserStore.Add(HttpKnownHeaderNames.TransferEncoding, TransferCodingHeaderParser.MultipleValueParser);
            parserStore.Add(HttpKnownHeaderNames.Upgrade, GenericHeaderParser.MultipleValueProductParser);
            parserStore.Add(HttpKnownHeaderNames.Via, GenericHeaderParser.MultipleValueViaParser);
            parserStore.Add(HttpKnownHeaderNames.Warning, GenericHeaderParser.MultipleValueWarningParser);
        }

        internal static void AddKnownHeaders(HashSet<string> headerSet)
        {
            Debug.Assert(headerSet != null);

            headerSet.Add(HttpKnownHeaderNames.CacheControl);
            headerSet.Add(HttpKnownHeaderNames.Connection);
            headerSet.Add(HttpKnownHeaderNames.Date);
            headerSet.Add(HttpKnownHeaderNames.Pragma);
            headerSet.Add(HttpKnownHeaderNames.Trailer);
            headerSet.Add(HttpKnownHeaderNames.TransferEncoding);
            headerSet.Add(HttpKnownHeaderNames.Upgrade);
            headerSet.Add(HttpKnownHeaderNames.Via);
            headerSet.Add(HttpKnownHeaderNames.Warning);
        }

        internal void AddSpecialsFrom(HttpGeneralHeaders sourceHeaders)
        {
            // Copy special values, but do not overwrite
            bool? chunked = TransferEncodingChunked;
            if (!chunked.HasValue)
            {
                TransferEncodingChunked = sourceHeaders.TransferEncodingChunked;
            }

            bool? close = ConnectionClose;
            if (!close.HasValue)
            {
                ConnectionClose = sourceHeaders.ConnectionClose;
            }
        }
    }
}


text:
// BridgeDb,
// An abstraction layer for identifier mapping services, both local and online.
//
// Copyright 2006-2009  BridgeDb developers
// Copyright 2012-2013  Christian Y. A. Brenninkmeijer
// Copyright 2012-2013  OpenPhacts
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package org.bridgedb.uri.api;

import java.util.HashSet;
import java.util.Set;

/**
 * Holder class for the main Meta Data of the relevant mappings in a single Set plus some set data.
 *
 * Does not include everything in the void header but only what is captured in the SQL.
 * @author Christian
 */
public class SetMappings {
    private final String id;
    private final String predicate;
    private final String justification;
    private final String mappingSource;
    private final Set<UriMapping> mappings;

    public SetMappings(String id, String predicate, String justification, String mappingSource){
        this.id = id;
        this.predicate = predicate;
        this.justification = justification;
        this.mappingSource = mappingSource;
        this.mappings = new HashSet<UriMapping>();
    }

    public void addMapping(UriMapping mapping){
        getMappings().add(mapping);
    }

    /**
     * @return the intId
     */
    public String getId() {
        return id;
    }

    /**
     * @return the predicate
     */
    public String getPredicate() {
        return predicate;
    }

    /**
     * @return the justification
     */
    public String getJustification() {
        return justification;
    }

    /**
     * @return the mappingSource
     */
    public String getMappingSource() {
        return mappingSource;
    }

    /**
     * @return the mappings
     */
    public Set<UriMapping> getMappings() {
        return mappings;
    }

    HashSet<String> getTargetUris() {
        HashSet<String> targetUris = new HashSet<String>();
        for (UriMapping mapping:mappings){
            targetUris.add(mapping.getTargetUri());
        }
        return targetUris;
    }

    void append(StringBuilder sb) {
        sb.append("\n\tid: ");
        sb.append(id);
        sb.append("\n\tpredicate: ");
        sb.append(predicate);
        sb.append("\n\tjustification: ");
        sb.append(justification);
        sb.append("\n\tmappingSource: ");
        sb.append(mappingSource);
        for (UriMapping mapping:mappings){
            mapping.append(sb);
        }
    }

  }


text:
package org.robbins.flashcards.repository.util;

import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
import java.util.Collection;
import java.util.Set;

import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.PersistenceUnitUtil;

import org.apache.commons.beanutils.PropertyUtils;
import org.robbins.flashcards.exceptions.RepositoryException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

@Component
public class FieldInitializerUtil {

    private static final Logger LOGGER = LoggerFactory.getLogger(FieldInitializerUtil.class);

    @PersistenceContext
    private EntityManager em;

    // Initialize the current 'field'
    // This is needed since Hibernate will not auto-initialize most
    // collections
    // Therefore, if we want to return the field in the response, we
    // need to make sure it is loaded
    private void initializeField(final Object entity, final String field) {
        final PersistenceUnitUtil unitUtil = em.getEntityManagerFactory().getPersistenceUnitUtil();

        try {
            // if the requested 'field' exists and hasn't been
            if ((PropertyUtils.isReadable(entity, field))
                    && (!unitUtil.isLoaded(entity, field))) {
                final Object value = PropertyUtils.getProperty(entity, field);
                // is the 'field' a collection?
                if (value instanceof Collection<?>) {
                    LOGGER.trace("Initializing collection: {}", field);

                    // this is why we are here!
                    // initialize the collection
                    initializeCollection((Collection<?>) value);
                }
            }
        } catch (IllegalAccessException | InvocationTargetException | NoSuchMethodException e) {
            e.printStackTrace();
            throw new RepositoryException(e.getMessage(), e);
        }
    }

    public void initializeEntity(final Collection<?> entities, final Set<String> fields)
            throws RepositoryException {
        entities.forEach(entity -> initializeEntity(entity, fields));
    }

    public void initializeEntity(final Object[] entities, final Set<String> fields)
            throws RepositoryException {
        Arrays.stream(entities).forEach(entity -> initializeEntity(entity, fields));
    }

    // make sure each of the requested 'fields' are initialized by Hibernate
    public void initializeEntity(final Object entity, final Set<String> fields)
            throws RepositoryException {

        if (entity == null) return;

        fields.forEach(field -> initializeField(entity, field));
    }

    private void initializeCollection(final Collection<?> collection) {
        if (collection == null) return;

        // we force the ORM to initialize the collection by touching it (hasNext)
        collection.iterator().hasNext();
    }
}


text:
import datetime

# pypi
from six import text_type

# ==============================================================================

class ApiError(Exception):
    """
    Base class for handling the Facebook API errors.
    """

    code = None
    type = None
    message = None
    response = None
    raised = None

    def __init__(self, code=None, type=None, message=None, response=None, raised=None):
        self.code = code
        self.type = type
        self.message = message
        self.response = response
        self.raised = raised

    def __str__(self):
        return text_type("ApiError: {code} | {type} | {message}").format(
            code=self.code, type=self.type, message=self.message
        )

class ApiAuthError(ApiError):
    """
    Facebook api authentification error
    """

    pass

class ApiAuthExpiredError(ApiAuthError):
    """
    Facebook api authentification error due to expiry
    """

    pass

class ApiApplicationError(ApiError):
    """
    Application set up error
    """

    pass

class ApiResponseError(ApiError):
    """
    Api Response Error
    """

    pass

class ApiRuntimeError(ApiError):
    """
    Runtime Error
    """

    pass

class ApiRuntimeVerirficationFormatError(ApiRuntimeError):
    """
    Raised if there is an error on the applicaiton when run: Invalid verification code format
    """

    pass

class ApiRuntimeGrantError(ApiRuntimeError):
    """
    Raised if there is an error on the application when run: Invalid verification code format
    """

    pass

class ApiRuntimeScopeError(ApiRuntimeError):
    """
    Raised if there is an error on the application when run: Invalid verification code format
    """

    pass

class ApiRuntimeGraphMethodError(ApiError):
    """
    Raised if there is an error on the application when run: Invalid graph method
    """

    pass

class ApiRatelimitedError(ApiError):
    """
    Raised if the application is ratelimited

    Select bits of text from Facebook's publicly shared documents are

    The following text is copyright Facebook and appears on the url:
        https://developers.facebook.com/docs/graph-api/advanced/rate-limiting/

    -- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- -
    All responses to calls made to the Graph API include an X-App-Usage HTTP header. This header contains the current percentage of usage for your app. This percentage is equal to the usage shown to you in the rate limiting graphs. Use this number to dynamically balance your call load to avoid being throttled.

    The rate limiting header is a JSON-formatted string in the following form:

        {
          "call_count"    : x,
          "total_time"    : y,
          "total_cputime" : z
        }

    The values for x, y and z are whole numbers representing the percentage used values for each of the metrics. When any of these metrics exceed 100 the app is rate limited.

    What Errors Will My App See?

    Throttling Type                 At least                    Error Code
    Application-level throttling    200 calls/person/hour       4
    Account-level throttling        Not applicable              17
    Page-level throttling           4800 calls/person/24-hours  32
    Custom-level throttling         Not applicable              613

    -- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- - - ---- -

    From Facebook's Application Dashboard, also copyright Facebook:

    Rate limiting defines limits on how many API calls can be made within a specified time period. Application-level rate limits apply to calls made using any access token other than a Page access token and ads APIs calls. The total number of calls your app can make per hour is 200 times the number of users. Please note this isn't a per-user limit. Any individual user can make more than 200 calls per hour, as long as the total for all users does not exceed the app maximum.
    """

    pass

class ApiUnhandledError(ApiError):
    """
    Raised if something bad happened, so you only have to track one error.
    Note that this inherits from ApiError - so this should be the first thing you catch

    Good - raises ApiUnhandledError
        try:
            raise ApiUnhandledError()
        except ApiUnhandledError, e:
            print "raised ApiUnhandledError"
        except ApiError, e:
            print "raised ApiError"

    Bad - raises ApiError
        try:
            raise ApiUnhandledError()
        except ApiError, e:
            print "raised ApiError"
        except ApiUnhandledError, e:
            print "raised ApiUnhandledError"

    """

    def __str__(self):
        return "ApiError: %s " % (self.raised)

class AuthenticatedHubRequired(Exception):
    """
    raised when an unauthenticated hub tries to perform an authenticated function
    """

    pass

def reformat_error(json_string, raised=None):

    rval = {"message": None, "type": None, "code": None, "raised": None}

    for k in rval.keys():
        if k in json_string:
            rval[k] = json_string[k]
    if raised is not None:
        rval["raised"] = raised
    return rval

def facebook_time(fb_time):
    """parses Facebook's timestamp into a datetime object"""
    return datetime.datetime.strptime(fb_time, "%Y-%m-%dT%H:%M:%S+0000")


text:
package com.KST.eCommerce;

/**
 *
 * @author Kureishi Shivanand
 */
public class DummyPaymentProcessor extends PaymentProcessor {
    // Overview: Prints message at the end of the session

    /**
     * Prints processing message
     *
     * @return String
     */
    public String processPayment() {
        // EFFECTS: Returns the message to indicate local session is ending

        return "Payment recieved.";
    }
}


text:
require 'spec_helper'

describe 'atom' do
  let(:facts) { default_test_facts }
  let(:params) { {
    :packages => ['package-1', 'package-2'],
    :themes   => ['theme-1', 'theme-2']
  } }

  it do
    should include_class('brewcask')

    should contain_package('atom').with({
      :ensure          => 'present',
      :provider        => 'brewcask',
      :install_options => ['--appdir=/Applications', '--binarydir=/test/boxen/bin'],
    })

    %w(package-1 package-2 theme-1 theme-2).each do |pkg|
      should contain_package(pkg).with({
        :ensure   => 'latest',
        :provider => 'apm'
      })
    end
  end
end


text:
package com.mossle.internal.mail.client;

import java.io.*;

import java.net.*;

import java.util.*;

import com.mossle.core.mapper.JsonMapper;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.beans.factory.annotation.Value;

import org.springframework.stereotype.Component;

public class HttpMailClient implements MailClient {
    private static Logger logger = LoggerFactory
            .getLogger(HttpMailClient.class);
    private String baseUrl;
    private String configCode;
    private JsonMapper jsonMapper = new JsonMapper();

    public void sendMail(String to, String templateCode,
            Map<String, Object> parameter) throws Exception {
        HttpURLConnection conn = (HttpURLConnection) new URL(baseUrl)
                .openConnection();
        conn.setDoOutput(true);
        conn.setDoInput(true);

        String queryString = "to=" + to + "&configCode=" + configCode
                + "&templateCode=" + templateCode + "&data="
                + jsonMapper.toJson(parameter);
        conn.getOutputStream().write(queryString.getBytes("UTF-8"));
        conn.getOutputStream().flush();

        InputStream is = conn.getInputStream();
        int len = -1;
        byte[] b = new byte[1024];

        ByteArrayOutputStream baos = new ByteArrayOutputStream();

        while ((len = is.read(b, 0, 1024)) != -1) {
            baos.write(b, 0, len);
        }

        is.close();

        String text = new String(baos.toByteArray(), "UTF-8");
        logger.info("mail result : {}", text);
    }

    @Value("${mail.baseUrl}")
    public void setBaseUrl(String baseUrl) {
        this.baseUrl = baseUrl;
    }

    @Value("${mail.configCode}")
    public void setConfigCode(String configCode) {
        this.configCode = configCode;
    }
}


text:
class Cms::CheckLinks::Error::Page < Cms::CheckLinks::Error::Base
  belongs_to :page, class_name: "Cms::Page"

  set_permission_name "cms_check_links_errors"

  def content
    page
  end

  def private_show_path(*args)
    options = args.extract_options!
    options.merge!(site: (cur_site || site), report_id: report.id, id: id)
    helper_mod = Rails.application.routes.url_helpers
    helper_mod.cms_check_links_report_page_path(*args, options) rescue nil
  end

  class << self
    def content_name
      I18n.t("ss.page")
    end
  end
end


text:
<!doctype html>

    <meta charset="utf-8">
    <meta http-equiv="x-ua-compatible" content="ie=edge">
    <title>MERN Stack</title>
    <meta name="description" content="just another react + webpack boilerplate">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://cdn.rawgit.com/grvpanchal/bootstrap-essentials/v0.5.0/dist/css/bootstrap-essentials.min.css">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/animatecss/3.5.1/animate.min.css">
    <link href="/dist/themes/apple/css/apple-bootstrap.css" rel="stylesheet" id="bs-css">


text:
                        * <?php echo "is " . ucwords("mandatory field"); ?>

                    <form id="exportform" class="form-horizontal validate" action="#" method="post">

                            Module *

                                <select id="export_module" class="form-control" name="module_name" >
                                    <option value="">Select</option>
                                    <option value="<?php echo base_url('admin/export_csv/admission_type'); ?>">Admission Type</option>
                                    <option value="<?php echo base_url('admin/export_csv/batch'); ?>">Batch</option>
                                    <option value="<?php echo base_url('admin/export_csv/course'); ?>">Branch</option>
                                    <option value="<?php echo base_url('admin/export_csv/degree'); ?>">Department</option>
                                    <option value="<?php echo base_url('admin/export_csv/exam_manager'); ?>">Exam Manager</option>
                                    <option value="exam_marks">Exam Marks</option>
                                    <option value="<?php echo base_url('admin/export_csv/event_manager'); ?>">Event Manager</option>
                                    <option value="<?php echo base_url('admin/export_csv/fees_structure'); ?>">Fee Structure</option>
                                    <option value="<?php echo base_url('admin/export_csv/project_manager'); ?>">Project Manager</option>
                                    <option value="<?php echo base_url('admin/export_csv/semester'); ?>">Semester</option>
                                    <option value="<?php echo base_url('admin/export_csv/student'); ?>">Student</option>
                                    <option value="<?php echo base_url('admin/export_csv/subject'); ?>">Subject</option>
                                    <option value="<?php echo base_url('admin/export_csv/system_setting'); ?>">System Settings</option>
                                </select>

                            Department

                                <select id="degree" name="degree" class="form-control">
                                    <option value="">Select</option>
                                    <?php foreach ($degree as $row) { ?>
                                        <option value="<?php echo $row->d_id; ?>"><?php echo $row->d_name; ?></option>
                                    <?php } ?>
                                </select>

                            Branch

                                <select id="course" name="course" class="form-control">

                                </select>

                            Batch

                                <select id="batch" name="batch" class="form-control">

                                </select>

                            Semester

                                <select id="semester" name="semester" class="form-control">
                                    <option value="">Select</option>
                                    <?php foreach ($semester as $row) { ?>
                                        <option value="<?php echo $row->s_id; ?>"><?php echo $row->s_name; ?></option>
                                    <?php } ?>
                                </select>

                            Exam

                                <select id="exam" name="exam" class="form-control">

                                </select>

                                <button id="export" type="submit" class="btn btn-info vd_bg-green"><?php echo ucwords("export"); ?></button>

                    </form>