text:
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.h0tk3y.betterParse/better-parse/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.github.h0tk3y.betterParse/better-parse)
[![Gradle build](https://github.com/h0tk3y/better-parse/workflows/Gradle%20build/badge.svg) ](https://github.com/h0tk3y/better-parse/actions?query=workflow%3A%22Gradle+build%22)

A nice parser combinator library for Kotlin JVM, JS, and Multiplatform projects

```kotlin
val booleanGrammar = object : Grammar<BooleanExpression>() {
    val id by regexToken("\\w+")
    val not by literalToken("!")
    val and by literalToken("&")
    val or by literalToken("|")
    val ws by regexToken("\\s+", ignore = true)
    val lpar by literalToken("(")
    val rpar by literalToken(")")

    val term by
        (id use { Variable(text) }) or
        (-not * parser(this::term) map { Not(it) }) or
        (-lpar * parser(this::rootParser) * -rpar)

    val andChain by leftAssociative(term, and) { l, _, r -> And(l, r) }
    override val rootParser by leftAssociative(andChain, or) { l, _, r -> Or(l, r) }
}

val ast = booleanGrammar.parseToEnd("a & !b | b & (!a | c)")
 ```

### Using with Gradle

```groovy
dependencies {
   implementation("com.github.h0tk3y.betterParse:better-parse:0.4.4")
}
```

With multiplatform projects, it's OK to add the dependency just to the `commonMain` source set, or some other source set if you want it for specific parts of the code.

## Tokens ##
As many other language recognition tools, `better-parse` abstracts away from raw character input by
pre-processing it with a `Tokenizer`, that can match `Token`s (with regular expressions, literal values or arbitrary
against an input character sequence.

There are several kinds of supported `Token`s:

* a `regexToken("(?:my)?(?:regex))` is matched as a regular expression;
* a `literalToken("foo")` is matched literally, character to character;
* a `token { (charSequence, from) -> ... }` is matched using the passed function.

A `Tokenizer` tokenizes an input sequence such as `InputStream` or a `String` into a `Sequence<TokenMatch>`, providing
each with a position in the input.

One way to create a `Tokenizer` is to first define the `Tokens` to be matched:

```kotlin
val id = regexToken("\\w+")
val cm = literalToken(",")
val ws = regexToken("\\s+", ignore = true)
```

> A `Token` can be ignored by setting its `ignore = true`. An ignored token can still be matched explicitly, but if
another token is expected, the ignored one is just dropped from the sequence.

```kotlin
val tokenizer = DefaultTokenizer(listOf(id, cm, ws))
```

> Note: the tokens order matters in some cases, because the tokenizer tries to match them in exactly this order.
> For instance, if `literalToken("a")`
> is listed before `literalToken("aa")`, the latter will never be matched. Be careful with keyword tokens!
> If you match them with regexes, a word boundary `\b` in the end may help against ambiguity.

```kotlin
val tokenMatches: Sequence<TokenMatch> = tokenizer.tokenize("hello, world")
```

> A more convenient way of defining tokens is described in the [**Grammar**](#grammar) section.

It is possible to provide a custom implementation of a `Tokenizer`.

## Parser ##

A `Parser<T>` is an object that accepts an input sequence (`TokenMatchesSequence`) and
tries to convert some (from none to all) of its items into a `T`. In `better-parse`, parsers are also
the building blocks used to create new parsers by *combining* them.

When a parser tries to process the input, there are two possible outcomes:

* If it succeeds, it returns `Parsed<T>` containing the `T` result and the `nextPosition: Int` that points to what
it left unprocessed. The latter can then be, and often is, passed to another parser.

* If it fails, it reports the failure returning an `ErrorResult`, which provides detailed information about the failure.

A very basic parser to start with is a `Token` itself: given an input `TokenMatchesSequence` and a position in it,
it succeeds if the sequence starts with the match of this token itself
_(possibly, skipping some **ignored** tokens)_ and returns that `TokenMatch`, pointing at the next token
with the `nextPosition`.

```kotlin
val a = regexToken("a+")
val b = regexToken("b+")
val tokenMatches = DefaultTokenizer(listOf(a, b)).tokenize("aabbaaa")
val result = a.tryParse(tokenMatches, 0) // contains the match for "aa" and the next index is 1 for the match of b
```

## Combinators ##

Simpler parsers can be combined to build a more complex parser, from tokens to terms and to the whole language.
There are several kinds of combinators included in `better-parse`:

* `map`, `use`, `asJust`

    The map combinator takes a successful input of another parser and applies a transforming function to it.
    The error results are returned unchanged.

    ```kotlin
    val id = regexToken("\\w+")
    val aText = a map { it.text } // Parser<String>, returns the matched text from the input sequence
    ```

    A parser for objects of a custom type can be created with `map`:

    ```kotlin
    val variable = a map { JavaVariable(name = it.text) } // Parser<JavaVariable>.
    ```

    * `someParser use { ... }` is a `map` equivalent that takes a function with receiver instead. Example: `id use { text }`.

    * `foo asJust bar` can be used to map a parser to some constant value.

* `optional(...)`

     Given a `Parser<T>`, tries to parse the sequence with it, but returns a `null` result if the parser failed, and thus never fails itself:

     ```kotlin
     val p: Parser<T> = ...
     val o = optional(p) // Parser<T?>
     ```

* `and`, `and skip(...)`

    The tuple combinator arranges the parsers in a sequence, so that the remainder of the first one goes to the second one and so on.
    If all the parsers succeed, their results are merged into a `Tuple`. If either parser failes, its `ErrorResult` is returned by the combinator.

    ```kotlin
    val a: Parser<A> = ...
    val b: Parser<B> = ...
    val aAndB = a and b                 // This is a `Parser<Tuple2<A, B>>`
    val bAndBAndA = b and b and a       // This is a `Parser<Tuple3<B, B, A>>`
    ```

     You can `skip(...)` components in a tuple combinator: the parsers will be called just as well, but their results won't be included in the
     resulting tuple:

     ```kotlin
     val bbWithoutA = skip(a) and b and skip(a) and b and skip(a)  // Parser<Tuple2<B, B>>
     ```

     > If all the components in an `and` chain are skipped except for one `Parser<T>`, the resulting parser
      is `Parser<T>`, not `Parser<Tuple1<T>>`.

     To process the resulting `Tuple`, use the aforementioned `map` and `use`. These parsers are equivalent:

     * ```val fCall = id and skip(lpar) and id and skip(rpar) map { (fName, arg) -> FunctionCall(fName, arg) }```

     * ```val fCall = id and lpar and id and rpar map { (fName, _, arg, _) -> FunctionCall(fName, arg) }```

     * ```val fCall = id and lpar and id and rpar use { FunctionCall(t1, t3) }```

     * ```val fCall = id * -lpar * id * -rpar use { FunctionCall(t1, t2) }``` (see operators below)

     > There are `Tuple` classes up to `Tuple16` and the corresponding `and` overloads.

     ##### Operators

     There are operator overloads for more compact `and` chains definition:

     * `a * b` is equivalent to `a and b`.

     * `-a` is equivalent to `skip(a)`.

     With these operators, the parser `a and skip(b) and skip(c) and d` can also be defined as
     `a * -b * -c * d`.

 * `or`

     The alternative combinator tries to parse the sequence with the parsers it combines one by one until one succeeds. If all the parsers fail,
     the returned `ErrorResult` is an `AlternativesFailure` instance that contains all the failures from the parsers.

     The result type for the combined parsers is the least common supertype (which is possibly `Any`).

     ```kotlin
     val expr = const or variable or fCall
     ```

  * `zeroOrMore(...)`, `oneOrMore(...)`, `N times`, `N timesOrMore`, `N..M times`

      These combinators transform a `Parser<T>` into a `Parser<List<T>>`, invokng the parser several times and failing if there was not
      enough matches.

      ```kotlin
      val modifiers = zeroOrMore(functionModifier)
      val rectangleParser = 4 times number map { (a, b, c, d) -> Rect(a, b, c, d) }
      ```

  * `separated(term, separator)`, `separatedTerms(term, separator)`, `leftAssociative(...)`, `rightAssociative(...)`

      Combines the two parsers, invoking them in turn and thus parsing a sequence of `term` matches separated by `separator` matches.

      The result is a `Separated<T, S>` which provides the matches of both parsers (note that terms are one more than separators) and
      can also be reduced in either direction.

      ```kotlin
      val number: Parser<Int> = ...
      val sumParser = separated(number, plus) use { reduce { a, _, b -> a + b } }
      ```

      The `leftAssociative` and `rightAssociative` combinators do exactly this, but they take the reducing operation as they are built:

      ```kotlin
      val term: Parser<Term>
      val andChain = leftAssociative(term, andOperator) { l, _, r -> And(l, r) }
      ```

## Grammar

As a convenient way of defining a grammar of a language, there is an abstract class `Grammar`, that collects the `by`-delegated
properties into a `Tokenizer` automatically, and also behaves as a composition of the `Tokenizer` and the `rootParser`.

*Note:* a `Grammar` also collects `by`-delegated `Parser<T>` properties so that they can be accessed as
`declaredParsers` along with the tokens. As a good style, declare the parsers inside a `Grammar` by delegation as well.

```kotlin
interface Item
class Number(val value: Int) : Item
class Variable(val name: String) : Item

class ItemsParser : Grammar<List<Item>>() {
    val num by regexToken("\\d+")
    val word by regexToken("[A-Za-z]+")
    val comma by regexToken(",\\s+")

    val numParser by num use { Number(text.toInt()) }
    val varParser by word use { Variable(text) }

    override val rootParser by separatedTerms(numParser or varParser, comma)
}

val result: List<Item> = ItemsParser().parseToEnd("one, 2, three, 4, five")
```

To use a parser that has not been constructed yet, reference it with `parser { someParser }` or `parser(this::someParser)`:

```kotlin
val term by
    constParser or
    variableParser or
    (-lpar and parser(this::term) and -rpar)
```

A `Grammar` implementation can override the `tokenizer` property to provide a custom implementation of `Tokenizer`.

## Syntax trees

A `Parser<T>` can be converted to another `Parser<SyntaxTree<T>>`, where a `SyntaxTree<T>`, along with the parsed `T`
contains the children syntax trees, the reference to the parser and the positions in the input sequence.
This can be done with `parser.liftToSyntaxTreeParser()`.

This can be used for syntax highlighting and inspecting the resulting tree in case the parsed result
does not contain the full syntactic structure.

For convenience, a `Grammar` can also be lifted to that parsing a `SyntaxTree` with
`grammar.liftToSyntaxTreeGrammar()`.

```kotlin
val treeGrammar = booleanGrammar.liftToSyntaxTreeGrammar()
val tree = treeGrammar.parseToEnd("a & !b | c -> d")
assertTrue(tree.parser == booleanGrammar.implChain)
val firstChild = tree.children.first()
assertTrue(firstChild.parser == booleanGrammar.orChain)
assertTrue(firstChild.range == 0..9)
```

There are optional arguments for customizing the transformation:

* `LiftToSyntaxTreeOptions`
  * `retainSkipped` — whether the resulting syntax tree should include skipped `and` components;
  * `retainSeparators` — whether the `Separated` combinator parsed separators should be included;
* `structureParsers` — defines the parsers that are retained in the syntax tree; the nodes with parsers that are
  not in this set are flattened so that their children are attached to their parents in their place.

  For `Parser<T>`, the default is `null`, which means no nodes are flattened.

  In case of `Grammar<T>`, `structureParsers` defaults to the grammar's `declaredParsers`.

* `transformer` — a strategy to transform non-built-in parsers. If you define your own combinators and want them
  to be lifted to syntax tree parsers, pass a `LiftToSyntaxTreeTransformer` that will be called on the parsers. When
  a custom combinator nests another parser, a transformer implementation should call `default.transform(...)` on that parser.

See [`SyntaxTreeDemo.kt`](https://github.com/h0tk3y/better-parse/blob/master/demo/demo-jvm/src/main/kotlin/com/example/SyntaxTreeDemo.kt) for an example of working with syntax trees.

## Examples

* A boolean expressions parser that constructs a simple AST: [`BooleanExpression.kt`](https://github.com/h0tk3y/better-parse/blob/master/demo/demo-jvm/src/main/kotlin/com/example/BooleanExpression.kt)
* An integer arithmetic expressions evaluator: [`ArithmeticsEvaluator.kt`](https://github.com/h0tk3y/better-parse/blob/master/demo/demo-jvm/src/main/kotlin/com/example/ArithmeticsEvaluator.kt)
* A toy programming language parser: [(link)](https://github.com/h0tk3y/compilers-course/blob/master/src/main/kotlin/com/github/h0tk3y/compilersCourse/parsing/Parser.kt)
* A sample JSON parser by [silmeth](https://github.com/silmeth): [(link)](https://github.com/silmeth/jsonParser)

## Benchmarks

See the benchmarks repository [`h0tk3y/better-parse-benchmark`](https://github.com/h0tk3y/better-parse-benchmark) and feel free to contribute.


text:
package sti

import (
	"log"
	"os"
	"path/filepath"

	"github.com/openshift/source-to-image/pkg/sti/docker"
	"github.com/openshift/source-to-image/pkg/sti/script"
	"github.com/openshift/source-to-image/pkg/sti/tar"
	"github.com/openshift/source-to-image/pkg/sti/util"
)

// requestHandler encapsulates dependencies needed to fulfill requests.
type requestHandler struct {
	request      *STIRequest
	result       *STIResult
	postExecutor postExecutor
	installer    script.Installer
	fs           util.FileSystem
	docker       docker.Docker
	tar          tar.Tar
}

type postExecutor interface {
	PostExecute(containerID string, cmd []string) error
}

// newRequestHandler returns a new handler for a given request.
func newRequestHandler(req *STIRequest) (*requestHandler, error) {
	if req.Verbose {
		log.Printf("Using docker socket: %s\n", req.DockerSocket)
	}

	docker, err := docker.NewDocker(req.DockerSocket, req.Verbose)
	if err != nil {
		return nil, err
	}

	return &requestHandler{
		request:   req,
		docker:    docker,
		installer: script.NewInstaller(req.BaseImage, req.ScriptsUrl, docker, req.Verbose),
		fs:        util.NewFileSystem(req.Verbose),
		tar:       tar.NewTar(req.Verbose),
	}, nil
}

func (h *requestHandler) setup(requiredScripts, optionalScripts []string) (err error) {
	if h.request.workingDir, err = h.fs.CreateWorkingDirectory(); err != nil {
		return err
	}

	h.result = &STIResult{
		Success:    false,
		WorkingDir: h.request.workingDir,
	}

	dirs := []string{"upload/scripts", "downloads/scripts", "downloads/defaultScripts"}
	for _, v := range dirs {
		if err = h.fs.MkdirAll(filepath.Join(h.request.workingDir, v)); err != nil {
			return err
		}
	}

	if err = h.installer.DownloadAndInstall(requiredScripts, h.request.workingDir, true); err != nil {
		return err
	}

	if err = h.installer.DownloadAndInstall(optionalScripts, h.request.workingDir, false); err != nil {
		return err
	}

	return nil
}

func (h *requestHandler) generateConfigEnv() (configEnv []string) {
	if len(h.request.Environment) > 0 {
		for key, val := range h.request.Environment {
			configEnv = append(configEnv, key+"="+val)
		}
	}
	return
}

func (h *requestHandler) execute(command string) error {
	if h.request.Verbose {
		log.Printf("Using image name %s", h.request.BaseImage)
	}

	uploadDir := filepath.Join(h.request.workingDir, "upload")
	tarFileName, err := h.tar.CreateTarFile(h.request.workingDir, uploadDir)
	if err != nil {
		return err
	}

	tarFile, err := h.fs.Open(tarFileName)
	if err != nil {
		return err
	}
	defer tarFile.Close()

	opts := docker.RunContainerOptions{
		Image:     h.request.BaseImage,
		Stdin:     tarFile,
		Stdout:    os.Stdout,
		Stderr:    os.Stderr,
		PullImage: true,
		Command:   command,
		Env:       h.generateConfigEnv(),
		PostExec:  h,
	}
	return h.docker.RunContainer(opts)
}

func (h *requestHandler) PostExecute(containerID string, cmd []string) (err error) {
	h.result.Success = true
	if h.postExecutor != nil {
		err = h.postExecutor.PostExecute(containerID, cmd)
		if err != nil {
			log.Printf("An error occurred in post executor: %v", err)
		}
	}
	return err
}

func (h *requestHandler) cleanup() {
	if h.request.PreserveWorkingDir {
		log.Printf("Temporary directory '%s' will be saved, not deleted\n", h.request.workingDir)
	} else {
		h.fs.RemoveDirectory(h.request.workingDir)
	}
}

redpajama_set_name: RedPajamaGithub


text:
using System;
using System.Linq;
using System.Web.Routing;

using CMS.Routing.Web;
using CMS.Tests;

using NSubstitute;
using NUnit.Framework;

namespace Kentico.Web.Mvc.Tests
{
    [TestFixture]
    [Category("Unit")]
    public class RouteCollectionAddRoutesMethodsTests
    {
        /// <summary>
        /// Dummy class for test purposes.
        /// </summary>
        private class TestHandler
        {
        };

        [Test]
        public void MapRoutes_ValidRoute_RouteGetsMapped()
        {
            var routes = Substitute.For<RouteCollection>();
            var handler = new RegisterHttpHandlerAttribute("TestRoute/SomeVirtualPath", typeof (TestHandler));
            handler.PreInit();

            routes.Kentico().MapRoutes();
            var registeredRoutes = HttpHandlerRouteTable.Default.GetRoutes().ToList();
            var actualRoutes = registeredRoutes.Where(i => i.Url == handler.RouteTemplate).ToList();

            CMSAssert.All(
                () => Assert.That(actualRoutes.Count, Is.EqualTo(1)),
                () => Assert.That(actualRoutes.First().Url, Is.EqualTo(handler.RouteTemplate)));
        }

        [Test]
        public void MapRoutes_NullExtensionPoint_ThrowsException()
        {
            Assert.That(() => ((ExtensionPoint<RouteCollection>)null).MapRoutes(),
                Throws.Exception
                    .TypeOf<ArgumentNullException>());
        }
    }
}


text:
<?php

namespace Zumba\GastonJS\Browser;

/**
 * Trait BrowserAuthenticationTrait
 * @package Zumba\GastonJS\Browser
 */
trait BrowserAuthenticationTrait {
  /**
   * Sets basic HTTP authentication
   * @param $user
   * @param $password
   * @return bool
   */
  public function setHttpAuth($user, $password) {
    return $this->command('set_http_auth', $user, $password);
  }
}


text:
{-#LANGUAGE OverloadedStrings #-}
{-#LANGUAGE NoImplicitPrelude #-}
module Web.Resourcerer.Serve
where

import Praglude
import Web.Resourcerer.Resource (Resource (..))
import Web.Resourcerer.Api ( runResource
                           , runApi
                           , methodFromBS
                           , ApiContext (..)
                           , ApiResponse (..)
                           , ApiError (..)
                           , PostedBody (..)
                           , Api)
import qualified Web.Resourcerer.Api as Api
import qualified Web.Resourcerer.Mime as Mime
import Network.Wai ( Application
                   , Request (..)
                   , Response
                   , responseLBS
                   , ResponseReceived
                   , lazyRequestBody
                   )
import Network.HTTP.Types ( status200
                          , status400
                          , status404
                          , status405
                          , status406
                          , status409
                          , Status
                          , queryToQueryText
                          )
import qualified Data.Aeson as JSON

resourceToApplication :: Resource -> Application
resourceToApplication resource rq respond = do
    let apiContext = waiRequestToApiContext rq def
        api = runResource resource
        accepts = apiContext ^. Api.accept
    apiResult <- runApi api apiContext
    (either catchApiError serveApiResponse apiResult) rq respond

serveApiResponse :: ApiResponse -> Application
serveApiResponse apiResponse rq respond = do
    let accepts = requestAccepts rq
    respond . apiResponseToWaiResponse accepts $ apiResponse

catchApiError :: ApiError -> Application
catchApiError err rq respond = do
    let (status, value) = apiErrorToResponse err
        accepts = requestAccepts rq
    respond $ structuredResponse accepts status value

requestAccepts :: Request -> [Mime.MimeType]
requestAccepts = Mime.parseAccept
               . fromMaybe "*/*"
               . lookup "Accept"
               . AList
               . requestHeaders

waiRequestToApiContext :: Request -> ApiContext -> ApiContext
waiRequestToApiContext rq context =
    let accepts = requestAccepts rq
        contentType = Mime.parse . fromMaybe "text/json" . lookup "Content-type" . AList . requestHeaders $ rq
    in context
        { _consumedPath = []
        , _remainingPath = pathInfo rq
        , _method = methodFromBS $ requestMethod rq
        , _accept = accepts
        , _queryParams = fromPairs . queryToQueryText $ queryString rq
        , _postedBody = do
            rawBody <- lazyRequestBody rq
            let parsedBody = parseRequestBody contentType rawBody
            return $ PostedBody rawBody contentType parsedBody
        }

parseRequestBody :: Mime.MimeType -> LByteString -> Maybe Value
parseRequestBody ty src =
    let matchingReaders = filter (\(k,v) -> Mime.isMatch k ty) structuredReaders
    in case matchingReaders of
        [] -> Nothing
        ((k, read):_) -> read src

type StructuredReader = LByteString -> Maybe Value

structuredReaders :: [(Mime.MimeType, StructuredReader)]
structuredReaders =
    [ ("text/json", decodeJSON)
    , ("application/json", decodeJSON)
    , ("text/plain", Just . JSON.String . s)
    ]

type StructuredWriter = Value -> LByteString

structuredWriters :: [(Mime.MimeType, StructuredWriter)]
structuredWriters =
    [ ("text/json;charset=utf8", encodeJSON)
    , ("application/json", encodeJSON)
    , ("text/plain;charset=utf8", s . jsonToText)
    ]

jsonToText :: Value -> Text
jsonToText (JSON.String txt) = txt
jsonToText JSON.Null = ""
jsonToText (JSON.Bool True) = "true"
jsonToText (JSON.Bool False) = "false"
jsonToText (JSON.Number n) = s . show $ n
jsonToText (JSON.Array xs) = concat . intersperse ", " . map jsonToText . toList $ xs
jsonToText (JSON.Object xs) = concat . intersperse ", " $
    [ k <> ": " <> jsonToText v
    | (k, v)
    <- pairs xs
    ]

structuredResponse :: [Mime.MimeType] -> Status -> Value -> Response
structuredResponse accepts status body =
    let candidateWriters = do
            accept <- accepts
            (available, writer) <- structuredWriters
            if Mime.isMatch accept available
                then [(available, writer)]
                else []
    in case candidateWriters of
        [] ->
            responseLBS
                status406
                [ ("Content-type", "text/plain") ]
                "Not Acceptable"
        ((ty, write):_) ->
            responseLBS
                status
                [ ("Content-type", Mime.pack ty) ]
                (write body)

apiResponseToWaiResponse :: [Mime.MimeType] -> ApiResponse -> Response
apiResponseToWaiResponse accepts (StructuredBody body) =
    structuredResponse accepts status200 body
apiResponseToWaiResponse accepts (BinaryBody ty body) =
    if any (Mime.isMatch ty) accepts
        then responseLBS status200 [("Content-type", Mime.pack ty)] body
        else structuredResponse accepts status406 "Not Acceptable"

apiErrorToResponse :: ApiError -> (Status, Value)
apiErrorToResponse DoesNotExistError = (status404, "Not Found")
apiErrorToResponse AlreadyExistsError = (status409, "Already Exists")
apiErrorToResponse UnsupportedOperationError = (status405, "Method Not Allowed")
apiErrorToResponse UnsupportedMediaTypeError = (status406, "Not Acceptable")
apiErrorToResponse InvalidRequestError = (status400, "Invalid Request")


text:
@interface oneBubbleViewController : LZBBaseViewController

@end


text:
package com.vdurmont.emoji;

import org.json.JSONArray;
import org.json.JSONObject;

import java.io.IOException;
import java.io.InputStream;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;

/**
 * Loads the emojis from a JSON database.
 *
 * @author Vincent DURMONT [vdurmont@gmail.com]
 */
public class EmojiLoader {
    /**
     * No need for a constructor, all the methods are static.
     */
    private EmojiLoader() {}

    /**
     * Loads a JSONArray of emojis from an InputStream, parses it and returns the associated list of {@link com.vdurmont.emoji.Emoji}s
     *
     * @param stream the stream of the JSONArray
     *
     * @return the list of {@link com.vdurmont.emoji.Emoji}s
     * @throws IOException if an error occurs while reading the stream or parsing the JSONArray
     */
    public static List<Emoji> loadEmojis(InputStream stream) throws IOException {
        JSONArray emojisJSON = new JSONArray(inputStreamToString(stream));
        List<Emoji> emojis = new ArrayList<Emoji>(emojisJSON.length());
        for (int i = 0; i < emojisJSON.length(); i++) {
            Emoji emoji = buildEmojiFromJSON(emojisJSON.getJSONObject(i));
            if (emoji != null) {
                emojis.add(emoji);
            }
        }
        return emojis;
    }

    private static String inputStreamToString(InputStream stream) {
        Scanner s = new Scanner(stream, "UTF-8").useDelimiter("\\A");
        return s.hasNext() ? s.next() : "";
    }

    protected static Emoji buildEmojiFromJSON(JSONObject json) throws UnsupportedEncodingException {
        if (!json.has("emoji")) {
            return null;
        }

        byte[] bytes = json.getString("emoji").getBytes("UTF-8");
        String description = null;
        if (json.has("description")) {
            description = json.getString("description");
        }
        boolean supportsFitzpatrick = false;
        if (json.has("supports_fitzpatrick")) {
            supportsFitzpatrick = json.getBoolean("supports_fitzpatrick");
        }
        List<String> aliases = jsonArrayToStringList(json.getJSONArray("aliases"));
        List<String> tags = jsonArrayToStringList(json.getJSONArray("tags"));
        return new Emoji(description, supportsFitzpatrick, aliases, tags, bytes);
    }

    private static List<String> jsonArrayToStringList(JSONArray array) {
        List<String> strings = new ArrayList<String>(array.length());
        for (int i = 0; i < array.length(); i++) {
            strings.add(array.getString(i));
        }
        return strings;
    }
}


text:
<?php

namespace Liip\ImagineBundle\DependencyInjection\Factory\Resolver;

use Symfony\Component\Config\Definition\Builder\ArrayNodeDefinition;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Definition;
use Symfony\Component\DependencyInjection\DefinitionDecorator;
use Symfony\Component\DependencyInjection\Reference;

class AwsS3ResolverFactory implements ResolverFactoryInterface
{
    /**
     * {@inheritdoc}
     */
    public function create(ContainerBuilder $container, $resolverName, array $config)
    {
        $awsS3ClientId = 'liip_imagine.cache.resolver.'.$resolverName.'.client';
        $awsS3ClientDefinition = new Definition('Aws\S3\S3Client');
        if (method_exists($awsS3ClientDefinition, 'setFactory')) {
            $awsS3ClientDefinition->setFactory(array('Aws\S3\S3Client', 'factory'));
        } else {
            // to be removed when dependency on Symfony DependencyInjection is bumped to 2.6
            $awsS3ClientDefinition->setFactoryClass('Aws\S3\S3Client');
            $awsS3ClientDefinition->setFactoryMethod('factory');
        }
        $awsS3ClientDefinition->addArgument($config['client_config']);
        $container->setDefinition($awsS3ClientId, $awsS3ClientDefinition);

        $resolverDefinition = new DefinitionDecorator('liip_imagine.cache.resolver.prototype.aws_s3');
        $resolverDefinition->replaceArgument(0, new Reference($awsS3ClientId));
        $resolverDefinition->replaceArgument(1, $config['bucket']);
        $resolverDefinition->replaceArgument(2, $config['acl']);
        $resolverDefinition->replaceArgument(3, array_replace($config['url_options'], $config['get_options']));
        $resolverDefinition->replaceArgument(4, $config['put_options']);
        $resolverId = 'liip_imagine.cache.resolver.'.$resolverName;
        $container->setDefinition($resolverId, $resolverDefinition);

        if (isset($config['cache_prefix'])) {
            $resolverDefinition->addMethodCall('setCachePrefix', array($config['cache_prefix']));
        }

        if ($config['proxies']) {
            $proxiedResolverId = 'liip_imagine.cache.resolver.'.$resolverName.'.proxied';

            $container->setDefinition($proxiedResolverId, $resolverDefinition);

            $proxyResolverDefinition = new DefinitionDecorator('liip_imagine.cache.resolver.prototype.proxy');
            $proxyResolverDefinition->replaceArgument(0, new Reference($proxiedResolverId));
            $proxyResolverDefinition->replaceArgument(1, $config['proxies']);

            $container->setDefinition($resolverId, $proxyResolverDefinition);
        }

        if ($config['cache']) {
            $cachedResolverId = 'liip_imagine.cache.resolver.'.$resolverName.'.cached';

            $container->setDefinition($cachedResolverId, $container->getDefinition($resolverId));

            $cacheResolverDefinition = new DefinitionDecorator('liip_imagine.cache.resolver.prototype.cache');
            $cacheResolverDefinition->replaceArgument(0, new Reference($config['cache']));
            $cacheResolverDefinition->replaceArgument(1, new Reference($cachedResolverId));

            $container->setDefinition($resolverId, $cacheResolverDefinition);
        }

        $container->getDefinition($resolverId)->addTag('liip_imagine.cache.resolver', array(
            'resolver' => $resolverName,
        ));

        return $resolverId;
    }

    /**
     * {@inheritdoc}
     */
    public function getName()
    {
        return 'aws_s3';
    }

    /**
     * {@inheritdoc}
     */
    public function addConfiguration(ArrayNodeDefinition $builder)
    {
        $builder
            ->children()
                ->scalarNode('bucket')->isRequired()->cannotBeEmpty()->end()
                ->scalarNode('cache')->defaultValue(false)->end()
                ->scalarNode('acl')->defaultValue('public-read')->cannotBeEmpty()->end()
                ->scalarNode('cache_prefix')->defaultValue(null)->end()
                ->arrayNode('client_config')
                    ->isRequired()
                    ->prototype('variable')
                        ->treatNullLike(array())
                    ->end()
                ->end()
                /* @deprecated Use `get_options` instead */
                ->arrayNode('url_options')
                    ->useAttributeAsKey('key')
                    ->prototype('scalar')->end()
                ->end()
                ->arrayNode('get_options')
                    ->useAttributeAsKey('key')
                    ->prototype('scalar')->end()
                ->end()
                ->arrayNode('put_options')
                    ->useAttributeAsKey('key')
                    ->prototype('scalar')->end()
                ->end()
                ->arrayNode('proxies')
                    ->defaultValue(array())
                    ->useAttributeAsKey('name')
                    ->prototype('scalar')->end()
                ->end()
            ->end()
        ;
    }
}


text:
{% block sitemenu-content-secondary %}
{% load nav %}

View

Edit

{% endblock %}


text:
name = 'Zed A. Shaw'
age = 35 # not a lie
height = 74 # inches
weight = 180 # lbs
inch_to_centimeter_conversion_rate = 2.54
pound_to_kg_conversion_rate = 0.453592
eyes = 'Blue'
teeth = 'White'
hair = 'Brown'

print "Lets talk about %s." % name
print "He's %d inches tall." % height
print "Which is %2f cm." % (height * inch_to_centimeter_conversion_rate)
print "He's %d pounds heavy." % weight
print "Which is %2f kg." % (weight * pound_to_kg_conversion_rate)
print "Which is also %+2.2f kg." % (weight * pound_to_kg_conversion_rate)
print "Actually that's not too heavy."
print "He's got %s eyes and %s hair." % (eyes, hair)
print "His teeth are usually %s depending on the coffee." % teeth

# this line is tricky, try to get it exactly right
print "If I add %d, %d, and %d I get %d." %(
    age, height, weight, age + height + weight
)


text:
window.addEventListener('load', function($scope, $mdMedia) {

    var rtc = new WebRTC({ audio: true, video: true}, false);

    var jarvis = null;
    var currentConnection = null;

    rtc.on('clientList', function(newClientList) {
        Object.keys(newClientList).forEach(function(k) {
            var client = newClientList[k];
            if (client.name !== 'J.A.R.V.I.S.') return;
            jarvis = client;
        });
    }).on('localStream', function() {
        if (currentConnection) currentConnection.close();
            currentConnection = rtc.call(jarvis.id);
    }).on('remoteStream', function(event) {
        document.getElementById('video').srcObject = event.stream;
    });

    document.getElementById('camera').addEventListener('componentchanged', function (evt) {
        if (evt.detail.name !== 'rotation') { return; }
        rtc.socket.emit('Move', evt.detail.newData); // x, y, z
    });

});


text:
// @flow
import type { Alert, AlertDatasetConfig, SelectedAlert } from 'types/alerts.types';
import type { AlertFeatureProperties, Coordinates, MapboxFeaturePressEvent } from 'types/common.types';

import React, { PureComponent } from 'react';

import { mapboxStyles } from './styles';
import MapboxGL from '@react-native-mapbox-gl/maps';
import { type FeatureCollection, type Point, featureCollection, point } from '@turf/helpers';
import i18n from 'i18next';
import _ from 'lodash';
import moment from 'moment';
import queryAlerts from 'helpers/alert-store/queryAlerts';
import generateUniqueID from 'helpers/uniqueId';
import { DATASETS, MAP_LAYER_INDEXES } from 'config/constants';
import { getNeighboursSelected } from 'helpers/map';
import kdbush from 'kdbush';

// eslint-disable-next-line import/no-unused-modules
export type AlertsIndex = {|
  +ids: Array<number>,
  +coords: Array<number>,
  +points: Array<Alert>,
  +nodeSize: number
|};

type Props = {|
  +areaId?: ?string,
  +isActive: boolean,
  +onPress?: ?(MapboxFeaturePressEvent<AlertFeatureProperties>) => any,
  +slug: 'umd_as_it_happens' | 'viirs',
  +reportedAlerts: $ReadOnlyArray<Coordinates>,
  +selectedAlerts: $ReadOnlyArray<SelectedAlert>,
  +timeframe: number,
  +timeframeUnit: 'days' | 'months'
|};

type FeatureCollectionState = {|
  +recentAlerts: FeatureCollection<Point>,
  +reportedAlerts: FeatureCollection<Point>,
  +otherAlerts: FeatureCollection<Point>
|};

type State = {|
  ...FeatureCollectionState,
  +alertsIndex: ?AlertsIndex,
  +alertsFromDb: Array<Alert>
|};

const initalState = {
  recentAlerts: featureCollection([]),
  reportedAlerts: featureCollection([]),
  otherAlerts: featureCollection([]),
  alertsIndex: null,
  alertsFromDb: []
};

/**
 * Displays the alerts corresponding to the specified dataset and other criteria
 */
export default class AlertDataset extends PureComponent<Props, State> {
  activeRequestId: ?string;
  datasets: {
    [string]: AlertDatasetConfig & {
      name: string,
      recencyTimestamp: number
    }
  };

  constructor(props: Props) {
    super(props);
    this.state = initalState;

    const now = moment();
    this.datasets = _.mapValues(DATASETS, config => ({
      ...config,
      name: i18n.t(config.nameKey),
      recencyTimestamp: now
        .clone()
        .subtract(config.recencyThreshold, 'days')
        .valueOf()
    }));
  }

  componentDidMount() {
    this._loadAlertsFromDb();
  }

  componentDidUpdate(prevProps: Props, prevState: State) {
    if (
      this.props.slug !== prevProps.slug ||
      this.props.isActive !== prevProps.isActive ||
      this.props.timeframe !== prevProps.timeframe ||
      this.props.timeframeUnit !== prevProps.timeframeUnit ||
      this.props.areaId !== prevProps.areaId
    ) {
      this._loadAlertsFromDb();
    } else if (
      this.props.selectedAlerts !== prevProps.selectedAlerts ||
      this.props.reportedAlerts !== prevProps.reportedAlerts
    ) {
      if (this.state.alertsFromDb?.length && this.state.alertsIndex?.nodeSize) {
        if (!this.props.isActive) {
          return;
        }
        // if alerts are already cached - only refresh alert properties
        const updatedAlertState = this._createFeaturesForAlerts(this.state.alertsFromDb, this.state.alertsIndex);

        this.setState({
          ...updatedAlertState
        });
      } else {
        this._loadAlertsFromDb();
      }
    }
  }

  /**
   * Updates component state so that it holds the alerts from the local DB matching the constraints imposed by the
   * component props
   */
  _loadAlertsFromDb = () => {
    const { areaId, isActive, slug, timeframe, timeframeUnit } = this.props;

    // Reset the data in state before retrieving the updated data
    this.setState(initalState);

    if (!isActive) {
      return;
    }

    try {
      const requestId = generateUniqueID();
      this.activeRequestId = requestId;
      const alertsFromDb = queryAlerts({
        areaId: areaId ?? undefined,
        dataset: slug,
        timeAgo: { max: timeframe, unit: timeframeUnit },
        limitAlerts: true,
        distinctLocations: true
      });

      const alertsIndex: AlertsIndex = kdbush(alertsFromDb, p => p.long, p => p.lat);
      const updatedAlertState = this._createFeaturesForAlerts(alertsFromDb, alertsIndex);
      if (requestId !== this.activeRequestId) {
        return;
      }

      this.setState({
        alertsFromDb,
        ...updatedAlertState,
        alertsIndex: alertsIndex
      });
    } catch (err) {
      console.warn(err);
    }
  };

  _getAlertProperties = (alert: Alert, selectedNeighbours: $ReadOnlyArray<Alert>): AlertFeatureProperties => {
    const { name, recencyTimestamp, iconPrefix } = this.datasets[alert.slug];
    const reported = this.props.reportedAlerts.some(
      coordinates => coordinates.latitude === alert.lat && coordinates.longitude === alert.long
    );
    const isRecent = alert.date > recencyTimestamp;
    const selected = this._isAlertSelected(alert);
    const alertInClusterSelected = selectedNeighbours.includes(alert);
    const icon = this._getAlertIconName(iconPrefix, isRecent, reported, alertInClusterSelected, selected);
    return {
      // need to pass these as strings as they are rounded in onShapeSourcePressed method.
      lat: '' + alert.lat,
      long: '' + alert.long,
      icon,
      date: alert.date,
      type: 'alert',
      name,
      reported,
      selected,
      clusterId: reported ? 'reported' : isRecent ? 'recent' : 'other',
      datasetId: alert.slug
    };
  };

  _isAlertSelected = (alertToCheck: Alert) => {
    return !!this.props.selectedAlerts.find(
      alert => alert.lat === alertToCheck.lat && alert.long === alertToCheck.long
    );
  };

  _getAlertIconName = (
    iconPrefix: string,
    recent: boolean,
    reported: boolean,
    alertInClusterSelected: boolean,
    selected: boolean
  ) => {
    if (selected) {
      return 'selected';
    }
    let iconName = iconPrefix;
    if (reported) {
      iconName += 'Reported';
    } else if (recent) {
      iconName += 'Recent';
    }

    if (alertInClusterSelected) {
      iconName += 'Selected';
    }
    return iconName;
  };

  _createFeaturesForAlerts = (alerts: Array<Alert>, alertsIndex: AlertsIndex): FeatureCollectionState => {
    const selectedNeighbours = getNeighboursSelected(alertsIndex, this.props.selectedAlerts, alerts);
    const alertFeatures = alerts.map((alert: Alert) => {
      const properties = this._getAlertProperties(alert, selectedNeighbours);
      return point([alert.long, alert.lat], properties);
    });
    const alertsGroupedByCluster = _.groupBy(alertFeatures, feature => feature.properties?.['clusterId']);
    return {
      recentAlerts: featureCollection(alertsGroupedByCluster.recent ?? []),
      reportedAlerts: featureCollection(alertsGroupedByCluster.reported ?? []),
      otherAlerts: featureCollection(alertsGroupedByCluster.other ?? [])
    };
  };

  render() {
    // As tempted as you may be to make this function conditionally return `null` based on the
    // isActive prop, this causes issues with enabling/disabling alert types on iOS so please don't!
    const { slug } = this.props;

    const { recentAlerts, reportedAlerts, otherAlerts } = this.state;
    const { color, colorReported, colorRecent } = this.datasets[slug] ?? {};

    return (
      <React.Fragment>
        {this.renderCluster(`${slug}ReportedAlerts`, colorReported, reportedAlerts)}
        {this.renderCluster(`${slug}RecentAlerts`, colorRecent, recentAlerts, slug === 'umd_as_it_happens')}
        {this.renderCluster(`${slug}OtherAlerts`, color, otherAlerts)}
      </React.Fragment>
    );
  }

  renderCluster = (
    clusterName: string,
    clusterColor: any,
    alerts: FeatureCollection<Point>,
    darkTextOnCluster: boolean = false
  ) => {
    const idShapeSource = `${clusterName}Source`;
    const idClusterCountSymbolLayer = `${clusterName}PointCount`;
    const idClusterCircleLayer = `${clusterName}ClusteredPoints`;
    const idAlertSymbolLayer = `${clusterName}AlertLayer`;

    const clusterCountStyle = {
      ...mapboxStyles.clusterCount,
      ...(darkTextOnCluster ? mapboxStyles.darkTextClusterCount : {})
    };

    return (
      <MapboxGL.ShapeSource
        id={idShapeSource}
        cluster
        clusterRadius={120}
        clusterMaxZoomLevel={15}
        shape={alerts}
        onPress={this.props.onPress}
      >
        <MapboxGL.SymbolLayer
          id={idClusterCountSymbolLayer}
          style={clusterCountStyle}
          layerIndex={MAP_LAYER_INDEXES.alerts}
        />
        <MapboxGL.CircleLayer
          id={idClusterCircleLayer}
          belowLayerID={idClusterCountSymbolLayer}
          filter={['has', 'point_count']}
          style={{ ...mapboxStyles.clusteredPoints, circleColor: clusterColor }}
          layerIndex={MAP_LAYER_INDEXES.alerts}
        />
        <MapboxGL.SymbolLayer
          id={idAlertSymbolLayer}
          filter={['!', ['has', 'point_count']]}
          style={mapboxStyles.alert}
          layerIndex={MAP_LAYER_INDEXES.alerts}
        />
      </MapboxGL.ShapeSource>
    );
  };
}


text:
using System;
using NUnit.Framework;
using Potato.Net.Shared.Test.Mocks;

namespace Potato.Net.Shared.Test {
    [TestFixture]
    public class PacketQueueTest {

        /// <summary>
        /// Tests that a packet is returned immediately from telling the queue it has been sent,
        /// inferring the packet is not queued and should be sent immediately.
        /// </summary>
        [Test]
        public void TestPacketSendImmediate() {

            PacketQueue queue = new PacketQueue();
            IPacketWrapper packet = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper poppedPacket = queue.PacketSend(packet);

            // Client would send to the server.

            Assert.AreEqual(packet, poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);
            Assert.AreEqual(0, queue.QueuedPackets.Count);
        }

        /// <summary>
        /// Tests that a packet will not be sent to the server if another packet
        /// is currently sent to the server and awaiting a response.
        /// </summary>
        [Test]
        public void TestPacketSendQueued() {

            PacketQueue queue = new PacketQueue();
            IPacketWrapper firstPacket = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper poppedPacket = queue.PacketSend(firstPacket);

            // Client would send to the server.

            Assert.AreEqual(firstPacket, poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);

            IPacketWrapper secondPacket = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            poppedPacket = queue.PacketSend(secondPacket);

            // Popped packet is null, client would essentially ignore it until later.

            Assert.IsNull(poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);
            Assert.AreEqual(1, queue.QueuedPackets.Count);
        }

        /// <summary>
        /// Tests that response to a request will remove from the outgoing list of packets.
        /// </summary>
        [Test]
        public void TestPacketReceivedRemoveFromOutgoing() {

            PacketQueue queue = new PacketQueue();

            IPacketWrapper sentPacket = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper recievedPacket = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Response,
                    RequestId = 1
                }
            };

            IPacketWrapper poppedPacket = queue.PacketSend(sentPacket);

            // Client would send to the server.

            Assert.AreEqual(sentPacket, poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);
            Assert.AreEqual(0, queue.QueuedPackets.Count);

            poppedPacket = queue.PacketReceived(recievedPacket);

            Assert.IsNull(poppedPacket);
            Assert.AreEqual(0, queue.OutgoingPackets.Count);
            Assert.AreEqual(0, queue.QueuedPackets.Count);
        }

        /// <summary>
        /// Tests that a packet will be removed from the outgoing packets and a new packet is dequeued.
        /// </summary>
        [Test]
        public void TestPacketReceivedRemovedAndPopped() {
            PacketQueue queue = new PacketQueue();

            IPacketWrapper firstPacketRequest = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper secondPacketRequest = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 2
                }
            };

            IPacketWrapper firstPacketResponse = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Response,
                    RequestId = 1
                }
            };

            queue.PacketSend(firstPacketRequest);
            queue.PacketSend(secondPacketRequest);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);
            Assert.AreEqual(1, queue.QueuedPackets.Count);

            IPacketWrapper poppedPacket = queue.PacketReceived(firstPacketResponse);
            Assert.AreEqual(secondPacketRequest, poppedPacket);

            queue.PacketSend(poppedPacket);

            Assert.AreEqual(1, queue.OutgoingPackets.Count);
            Assert.AreEqual(0, queue.QueuedPackets.Count);
        }

        /// <summary>
        /// Tests that a connection restart will be required if a packet has expired (2 minutes)
        /// </summary>
        [Test]
        public void TestRestartConnectionOnQueueFailureTruey() {
            PacketQueue queue = new PacketQueue();
            IPacketWrapper packet = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1,
                    Stamp = DateTime.Now.AddMinutes(-5)
                }
            };

            IPacketWrapper poppedPacket = queue.PacketSend(packet);

            Assert.AreEqual(packet, poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);

            Assert.IsTrue(queue.RestartConnectionOnQueueFailure());
        }

        /// <summary>
        /// Tests that no restart of a connection will be required if the queue contains
        /// no old packets.
        /// </summary>
        [Test]
        public void TestRestartConnectionOnQueueFailureFalsey() {
            PacketQueue queue = new PacketQueue();
            IPacketWrapper packet = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper poppedPacket = queue.PacketSend(packet);

            Assert.AreEqual(packet, poppedPacket);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);

            Assert.IsFalse(queue.RestartConnectionOnQueueFailure());
        }

        /// <summary>
        /// Tests that we can get the original request packet given a response packet tht exists
        /// </summary>
        [Test]
        public void TestGetRequestPacketExists() {
            PacketQueue queue = new PacketQueue();
            IPacketWrapper packetRequest = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper packetResponse = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Response,
                    RequestId = 1
                }
            };

            queue.PacketSend(packetRequest);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);

            IPacketWrapper fetchedRequestPacket = queue.GetRequestPacket(packetResponse);

            Assert.AreEqual(packetRequest, fetchedRequestPacket);
        }

        /// <summary>
        /// Tests that we get a null value back when no request has been requested
        /// from a response.
        /// </summary>
        [Test]
        public void TestGetRequestPacketDoesNotExists() {
            PacketQueue queue = new PacketQueue();
            IPacketWrapper packetRequest = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Request,
                    RequestId = 1
                }
            };

            IPacketWrapper packetResponse = new MockPacket() {
                Packet = {
                    Origin = PacketOrigin.Client,
                    Type = PacketType.Response,
                    RequestId = 2
                }
            };

            queue.PacketSend(packetRequest);
            Assert.AreEqual(1, queue.OutgoingPackets.Count);

            IPacketWrapper fetchedRequestPacket = queue.GetRequestPacket(packetResponse);

            Assert.IsNull(fetchedRequestPacket);
        }
    }
}


text:
package net.ripe.db.whois.query.planner;

import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import net.ripe.db.whois.common.dao.RpslObjectDao;
import net.ripe.db.whois.common.dao.RpslObjectInfo;
import net.ripe.db.whois.common.domain.ResponseObject;
import net.ripe.db.whois.common.rpsl.ObjectType;
import net.ripe.db.whois.common.rpsl.RpslObject;
import net.ripe.db.whois.query.domain.MessageObject;
import net.ripe.db.whois.query.query.Query;
import net.ripe.db.whois.query.support.Fixture;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;

import java.util.Arrays;

import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.hasSize;
import static org.junit.Assert.assertThat;
import static org.mockito.Mockito.*;

@RunWith(MockitoJUnitRunner.class)
public class GroupObjectTypesFunctionTest {
    private Query query;

    @Mock private PrimaryObjectDecorator decorator;
    @Mock private RpslObjectDao rpslObjectDao;

    private GroupObjectTypesFunction subject;

    @Before
    public void setUp() {
        query = Query.parse("foo");
        subject = new GroupObjectTypesFunction(rpslObjectDao, query, Sets.newHashSet(decorator));
    }

    @Test
    public void apply_messageObject() {
        final ResponseObject input = new MessageObject("");
        final Iterable<ResponseObject> responseObjects = subject.apply(input);
        final Iterable<ResponseObject> relatedObjects = subject.getGroupedAfter();

        verify(decorator, times(0)).appliesToQuery(query);

        assertThat(responseObjects, contains(input));
        assertThat(Lists.newArrayList(relatedObjects), hasSize(0));
    }

    @Test
    public void apply_rpslObject_no_match() {
        final ResponseObject input = RpslObject.parse("inetnum:10.0.0.0");

        when(decorator.appliesToQuery(query)).thenReturn(false);

        final Iterable<ResponseObject> responseObjects = subject.apply(input);
        final Iterable<ResponseObject> relatedObjects = subject.getGroupedAfter();

        verify(decorator, times(0)).decorate(query, (RpslObject) input);

        assertThat(responseObjects, contains(input));
        assertThat(Lists.newArrayList(relatedObjects), hasSize(0));
    }

    @Test
    public void apply_rpslObject() {
        Fixture.mockRpslObjectDaoLoadingBehavior(rpslObjectDao);

        final ResponseObject input = RpslObject.parse("inetnum:10.0.0.0");

        final RpslObjectInfo info1 = new RpslObjectInfo(1, ObjectType.IRT, "IRT");
        final ResponseObject result1 = RpslObject.parse("irt:irt");

        final RpslObjectInfo info2 = new RpslObjectInfo(2, ObjectType.MNTNER, "MNTNER");
        final ResponseObject result2 = RpslObject.parse("mntner:mntner");

        when(decorator.appliesToQuery(query)).thenReturn(true);
        when(decorator.decorate(query, (RpslObject) input)).thenReturn(Arrays.asList(info1, info2));
        when(rpslObjectDao.getById(1)).thenReturn((RpslObject) result1);
        when(rpslObjectDao.getById(2)).thenReturn((RpslObject) result2);

        final Iterable<ResponseObject> responseObjects = subject.apply(input);
        final Iterable<ResponseObject> relatedObjects = subject.getGroupedAfter();

        assertThat(responseObjects, contains(input));
        assertThat(relatedObjects, contains(result2, result1));
    }
}


text:
package java.util.zip;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.HashSet;

import libcore.util.CountingOutputStream;
import libcore.util.EmptyArray;

/**
 * Used to write (compress) data into zip files.
 *
 *
{@code ZipOutputStream} is used to write {@link ZipEntry}s to the underlying
 * stream. Output from {@code ZipOutputStream} can be read using {@link ZipFile}
 * or {@link ZipInputStream}.
 *
 *
While {@code DeflaterOutputStream} can write compressed zip file
 * entries, this extension can write uncompressed entries as well.
 * Use {@link ZipEntry#setMethod} or {@link #setMethod} with the {@link ZipEntry#STORED} flag.
 *
 *
Example

 *
Using {@code ZipOutputStream} is a little more complicated than {@link GZIPOutputStream}
 * because zip files are containers that can contain multiple files. This code creates a zip
 * file containing several files, similar to the {@code zip(1)} utility.
 *

 * OutputStream os = ...
 * ZipOutputStream zos = new ZipOutputStream(new BufferedOutputStream(os));
 * try {
 *     for (int i = 0; i < fileCount; ++i) {
 *         String filename = ...
 *         byte[] bytes = ...
 *         ZipEntry entry = new ZipEntry(filename);
 *         zos.putNextEntry(entry);
 *         zos.write(bytes);
 *         zos.closeEntry();
 *     }
 * } finally {
 *     zos.close();
 * }
 *

 */
public class ZipOutputStream extends DeflaterOutputStream implements ZipConstants {

    /**
     * Indicates deflated entries.
     */
    public static final int DEFLATED = 8;

    /**
     * Indicates uncompressed entries.
     */
    public static final int STORED = 0;

    private static final int ZIP_VERSION_2_0 = 20; // Zip specification version 2.0.

    private byte[] commentBytes = EmptyArray.BYTE;

    private final HashSet<String> entries = new HashSet<String>();

    private int defaultCompressionMethod = DEFLATED;

    private int compressionLevel = Deflater.DEFAULT_COMPRESSION;

    private ByteArrayOutputStream cDir = new ByteArrayOutputStream();

    private ZipEntry currentEntry;

    private final CRC32 crc = new CRC32();

    private long offset = 0;

    /** The charset-encoded name for the current entry. */
    private byte[] nameBytes;

    /** The charset-encoded comment for the current entry. */
    private byte[] entryCommentBytes;

    private static final byte[] ZIP64_PLACEHOLDER_BYTES =
            new byte[] { (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff };

    /**
     * Whether this zip file needs a Zip64 EOCD record / zip64 EOCD record locator. This
     * will be true if we wrote an entry whose size or compressed size was too large for
     * the standard zip format or if we exceeded the maximum number of entries allowed
     * in the standard format.
     */
    private boolean archiveNeedsZip64EocdRecord;

    /**
     * Whether the current entry being processed needs a zip64 extended info record. This
     * will be true if the entry is too large for the standard zip format or if the offset
     * to the start of the current entry header is greater than 0xFFFFFFFF.
     */
    private boolean currentEntryNeedsZip64;

    /**
     * Whether we force all entries in this archive to have a zip64 extended info record.
     * This of course implies that the {@code currentEntryNeedsZip64} and
     * {@code archiveNeedsZip64EocdRecord} are always {@code true}.
     */
    private final boolean forceZip64;

    /**
     * Constructs a new {@code ZipOutputStream} that writes a zip file to the given
     * {@code OutputStream}.
     *
     *
UTF-8 will be used to encode the file comment, entry names and comments.
     */
    public ZipOutputStream(OutputStream os) {
        this(os, false /* forceZip64 */);
    }

    /**
     * @hide for testing only.
     */
    public ZipOutputStream(OutputStream os, boolean forceZip64) {
        super(new CountingOutputStream(os), new Deflater(Deflater.DEFAULT_COMPRESSION, true));
        this.forceZip64 = forceZip64;
    }

    /**
     * Closes the current {@code ZipEntry}, if any, and the underlying output
     * stream. If the stream is already closed this method does nothing.
     *
     * @throws IOException
     *             If an error occurs closing the stream.
     */
    @Override
    public void close() throws IOException {
        // don't call super.close() because that calls finish() conditionally
        if (out != null) {
            finish();
            def.end();
            out.close();
            out = null;
        }
    }

    /**
     * Closes the current {@code ZipEntry}. Any entry terminal data is written
     * to the underlying stream.
     *
     * @throws IOException
     *             If an error occurs closing the entry.
     */
    public void closeEntry() throws IOException {
        checkOpen();
        if (currentEntry == null) {
            return;
        }
        if (currentEntry.getMethod() == DEFLATED) {
            super.finish();
        }

        // Verify values for STORED types
        if (currentEntry.getMethod() == STORED) {
            if (crc.getValue() != currentEntry.crc) {
                throw new ZipException("CRC mismatch");
            }
            if (currentEntry.size != crc.tbytes) {
                throw new ZipException("Size mismatch");
            }
        }

        long curOffset = LOCHDR;

        // Write the DataDescriptor
        if (currentEntry.getMethod() != STORED) {
            curOffset += EXTHDR;

            // Data descriptor signature and CRC are 4 bytes each for both zip and zip64.
            writeLongAsUint32(out, EXTSIG);
            writeLongAsUint32(out, currentEntry.crc = crc.getValue());

            currentEntry.compressedSize = def.getBytesWritten();
            currentEntry.size = def.getBytesRead();

            if (currentEntryNeedsZip64) {
                // We need an additional 8 bytes to store 8 byte compressed / uncompressed
                // sizes.
                curOffset += 8;
                writeLongAsUint64(out, currentEntry.compressedSize);
                writeLongAsUint64(out, currentEntry.size);
            } else {
                writeLongAsUint32(out, currentEntry.compressedSize);
                writeLongAsUint32(out, currentEntry.size);
            }
        }
        // Update the CentralDirectory
        // http://www.pkware.com/documents/casestudies/APPNOTE.TXT
        int flags = currentEntry.getMethod() == STORED ? 0 : ZipFile.GPBF_DATA_DESCRIPTOR_FLAG;
        // Since gingerbread, we always set the UTF-8 flag on individual files if appropriate.
        // Some tools insist that the central directory have the UTF-8 flag.
        // http://code.google.com/p/android/issues/detail?id=20214
        flags |= ZipFile.GPBF_UTF8_FLAG;
        writeLongAsUint32(cDir, CENSIG);
        writeIntAsUint16(cDir, ZIP_VERSION_2_0); // Version this file was made by.
        writeIntAsUint16(cDir, ZIP_VERSION_2_0); // Minimum version needed to extract.
        writeIntAsUint16(cDir, flags);
        writeIntAsUint16(cDir, currentEntry.getMethod());
        writeIntAsUint16(cDir, currentEntry.time);
        writeIntAsUint16(cDir, currentEntry.modDate);
        writeLongAsUint32(cDir, crc.getValue());

        if (currentEntry.getMethod() == DEFLATED) {
            currentEntry.setCompressedSize(def.getBytesWritten());
            currentEntry.setSize(def.getBytesRead());
            curOffset += currentEntry.getCompressedSize();
        } else {
            currentEntry.setCompressedSize(crc.tbytes);
            currentEntry.setSize(crc.tbytes);
            curOffset += currentEntry.getSize();
        }

        if (currentEntryNeedsZip64) {
            // Refresh the extended info with the compressed size / size before
            // writing it to the central directory.
            Zip64.refreshZip64ExtendedInfo(currentEntry);

            // NOTE: We would've written out the zip64 extended info locator to the entry
            // extras while constructing the local file header. There's no need to do it again
            // here. If we do, there will be a size mismatch since we're calculating offsets
            // based on the *current* size of the extra data and not based on the size
            // at the point of writing the LFH.
            writeLongAsUint32(cDir, Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE);
            writeLongAsUint32(cDir, Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE);
        } else {
            writeLongAsUint32(cDir, currentEntry.getCompressedSize());
            writeLongAsUint32(cDir, currentEntry.getSize());
        }

        curOffset += writeIntAsUint16(cDir, nameBytes.length);
        if (currentEntry.extra != null) {
            curOffset += writeIntAsUint16(cDir, currentEntry.extra.length);
        } else {
            writeIntAsUint16(cDir, 0);
        }

        writeIntAsUint16(cDir, entryCommentBytes.length); // Comment length.
        writeIntAsUint16(cDir, 0); // Disk Start
        writeIntAsUint16(cDir, 0); // Internal File Attributes
        writeLongAsUint32(cDir, 0); // External File Attributes
        if (currentEntryNeedsZip64) {
            writeLongAsUint32(cDir, Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE);
        } else {
            writeLongAsUint32(cDir, currentEntry.localHeaderRelOffset);
        }
        cDir.write(nameBytes);
        nameBytes = null;
        if (currentEntry.extra != null) {
            cDir.write(currentEntry.extra);
        }
        offset += curOffset;
        if (entryCommentBytes.length > 0) {
            cDir.write(entryCommentBytes);
            entryCommentBytes = EmptyArray.BYTE;
        }
        currentEntry = null;
        crc.reset();
        def.reset();
        done = false;
    }

    /**
     * Indicates that all entries have been written to the stream. Any terminal
     * information is written to the underlying stream.
     *
     * @throws IOException
     *             if an error occurs while terminating the stream.
     */
    @Override
    public void finish() throws IOException {
        // TODO: is there a bug here? why not checkOpen?
        if (out == null) {
            throw new IOException("Stream is closed");
        }
        if (cDir == null) {
            return;
        }
        if (entries.isEmpty()) {
            throw new ZipException("No entries");
        }
        if (currentEntry != null) {
            closeEntry();
        }

        int cdirEntriesSize = cDir.size();
        if (archiveNeedsZip64EocdRecord) {
            Zip64.writeZip64EocdRecordAndLocator(cDir, entries.size(), offset, cdirEntriesSize);
        }

        // Write Central Dir End
        writeLongAsUint32(cDir, ENDSIG);
        writeIntAsUint16(cDir, 0); // Disk Number
        writeIntAsUint16(cDir, 0); // Start Disk

        // Instead of trying to figure out *why* this archive needed a zip64 eocd record,
        // just delegate all these values to the zip64 eocd record.
        if (archiveNeedsZip64EocdRecord) {
            writeIntAsUint16(cDir, 0xFFFF); // Number of entries
            writeIntAsUint16(cDir, 0xFFFF); // Number of entries
            writeLongAsUint32(cDir, 0xFFFFFFFF); // Size of central dir
            writeLongAsUint32(cDir, 0xFFFFFFFF); // Offset of central dir;
        } else {
            writeIntAsUint16(cDir, entries.size()); // Number of entries
            writeIntAsUint16(cDir, entries.size()); // Number of entries
            writeLongAsUint32(cDir, cdirEntriesSize); // Size of central dir
            writeLongAsUint32(cDir, offset); // Offset of central dir
        }

        writeIntAsUint16(cDir, commentBytes.length);
        if (commentBytes.length > 0) {
            cDir.write(commentBytes);
        }
        // Write the central directory.
        cDir.writeTo(out);
        cDir = null;
    }

    /**
     * Writes entry information to the underlying stream. Data associated with
     * the entry can then be written using {@code write()}. After data is
     * written {@code closeEntry()} must be called to complete the writing of
     * the entry to the underlying stream.
     *
     * @param ze
     *            the {@code ZipEntry} to store.
     * @throws IOException
     *             If an error occurs storing the entry.
     * @see #write
     */
    public void putNextEntry(ZipEntry ze) throws IOException {
        if (currentEntry != null) {
            closeEntry();
        }

        // Did this ZipEntry specify a method, or should we use the default?
        int method = ze.getMethod();
        if (method == -1) {
            method = defaultCompressionMethod;
        }

        // If the method is STORED, check that the ZipEntry was configured appropriately.
        if (method == STORED) {
            if (ze.getCompressedSize() == -1) {
                ze.setCompressedSize(ze.getSize());
            } else if (ze.getSize() == -1) {
                ze.setSize(ze.getCompressedSize());
            }
            if (ze.getCrc() == -1) {
                throw new ZipException("STORED entry missing CRC");
            }
            if (ze.getSize() == -1) {
                throw new ZipException("STORED entry missing size");
            }
            if (ze.size != ze.compressedSize) {
                throw new ZipException("STORED entry size/compressed size mismatch");
            }
        }

        checkOpen();
        checkAndSetZip64Requirements(ze);

        nameBytes = ze.name.getBytes(StandardCharsets.UTF_8);
        checkSizeIsWithinShort("Name", nameBytes);
        entryCommentBytes = EmptyArray.BYTE;
        if (ze.comment != null) {
            entryCommentBytes = ze.comment.getBytes(StandardCharsets.UTF_8);
            // The comment is not written out until the entry is finished, but it is validated here
            // to fail-fast.
            checkSizeIsWithinShort("Comment", entryCommentBytes);
        }

        def.setLevel(compressionLevel);
        ze.setMethod(method);

        currentEntry = ze;
        currentEntry.localHeaderRelOffset = offset;
        entries.add(currentEntry.name);

        // Local file header.
        // http://www.pkware.com/documents/casestudies/APPNOTE.TXT
        int flags = (method == STORED) ? 0 : ZipFile.GPBF_DATA_DESCRIPTOR_FLAG;
        // Java always outputs UTF-8 filenames. (Before Java 7, the RI didn't set this flag and used
        // modified UTF-8. From Java 7, when using UTF_8 it sets this flag and uses normal UTF-8.)
        flags |= ZipFile.GPBF_UTF8_FLAG;
        writeLongAsUint32(out, LOCSIG); // Entry header
        writeIntAsUint16(out, ZIP_VERSION_2_0); // Minimum version needed to extract.
        writeIntAsUint16(out, flags);
        writeIntAsUint16(out, method);
        if (currentEntry.getTime() == -1) {
            currentEntry.setTime(System.currentTimeMillis());
        }
        writeIntAsUint16(out, currentEntry.time);
        writeIntAsUint16(out, currentEntry.modDate);

        if (method == STORED) {
            writeLongAsUint32(out, currentEntry.crc);

            if (currentEntryNeedsZip64) {
                // NOTE: According to the spec, we're allowed to use these fields under zip64
                // as long as the sizes are <= 4G (and omit writing the zip64 extended information header).
                //
                // For simplicity, we write the zip64 extended info here even if we only need it
                // in the central directory (i.e, the case where we're turning on zip64 because the
                // offset to this entries LFH is > 0xFFFFFFFF).
                out.write(ZIP64_PLACEHOLDER_BYTES);  // compressed size
                out.write(ZIP64_PLACEHOLDER_BYTES);  // uncompressed size
            } else {
                writeLongAsUint32(out, currentEntry.size);
                writeLongAsUint32(out, currentEntry.size);
            }
        } else {
            writeLongAsUint32(out, 0);
            writeLongAsUint32(out, 0);
            writeLongAsUint32(out, 0);
        }

        writeIntAsUint16(out, nameBytes.length);

        if (currentEntryNeedsZip64) {
            Zip64.insertZip64ExtendedInfoToExtras(currentEntry);
        }

        if (currentEntry.extra != null) {
            writeIntAsUint16(out, currentEntry.extra.length);
        } else {
            writeIntAsUint16(out, 0);
        }
        out.write(nameBytes);
        if (currentEntry.extra != null) {
            out.write(currentEntry.extra);
        }
    }

    private void checkAndSetZip64Requirements(ZipEntry entry) {
        final long totalBytesWritten = getBytesWritten();
        final long entriesWritten = entries.size();

        currentEntryNeedsZip64 = false;
        if (forceZip64) {
            currentEntryNeedsZip64 = true;
            archiveNeedsZip64EocdRecord = true;
            return;
        }

        // In this particular case, we'll write a zip64 eocd record locator and a zip64 eocd
        // record but we won't actually need zip64 extended info records for any of the individual
        // entries (unless they trigger the checks below).
        if (entriesWritten == 64*1024 - 1) {
            archiveNeedsZip64EocdRecord = true;
        }

        // Check whether we'll need to write out a zip64 extended info record in both the local file header
        // and the central directory. In addition, we will need a zip64 eocd record locator
        // and record to mark this archive as zip64.
        //
        // TODO: This is an imprecise check. When method != STORED it's possible that the compressed
        // size will be (slightly) larger than the actual size. How can we improve this ?
        if (totalBytesWritten > Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE ||
                (entry.getSize() > Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE)) {
            currentEntryNeedsZip64 = true;
            archiveNeedsZip64EocdRecord = true;
        }
    }

    /**
     * Sets the comment associated with the file being written. See {@link ZipFile#getComment}.
     * @throws IllegalArgumentException if the comment is >= 64 Ki encoded bytes.
     */
    public void setComment(String comment) {
        if (comment == null) {
            this.commentBytes = EmptyArray.BYTE;
            return;
        }

        byte[] newCommentBytes = comment.getBytes(StandardCharsets.UTF_8);
        checkSizeIsWithinShort("Comment", newCommentBytes);
        this.commentBytes = newCommentBytes;
    }

    /**
     * Sets the compression level to be used
     * for writing entry data.
     */
    public void setLevel(int level) {
        if (level < Deflater.DEFAULT_COMPRESSION || level > Deflater.BEST_COMPRESSION) {
            throw new IllegalArgumentException("Bad level: " + level);
        }
        compressionLevel = level;
    }

    /**
     * Sets the default compression method to be used when a {@code ZipEntry} doesn't
     * explicitly specify a method. See {@link ZipEntry#setMethod} for more details.
     */
    public void setMethod(int method) {
        if (method != STORED && method != DEFLATED) {
            throw new IllegalArgumentException("Bad method: " + method);
        }
        defaultCompressionMethod = method;
    }

    static long writeLongAsUint32(OutputStream os, long i) throws IOException {
        // Write out the long value as an unsigned int
        os.write((int) (i & 0xFF));
        os.write((int) (i >> 8) & 0xFF);
        os.write((int) (i >> 16) & 0xFF);
        os.write((int) (i >> 24) & 0xFF);
        return i;
    }

    static long writeLongAsUint64(OutputStream os, long i) throws IOException {
        int i1 = (int) i;
        os.write(i1 & 0xFF);
        os.write((i1 >> 8) & 0xFF);
        os.write((i1 >> 16) & 0xFF);
        os.write((i1 >> 24) & 0xFF);

        int i2 = (int) (i >> 32);
        os.write(i2 & 0xFF);
        os.write((i2 >> 8) & 0xFF);
        os.write((i2 >> 16) & 0xFF);
        os.write((i2 >> 24) & 0xFF);

        return i;
    }

    static int writeIntAsUint16(OutputStream os, int i) throws IOException {
        os.write(i & 0xFF);
        os.write((i >> 8) & 0xFF);
        return i;
    }

    /**
     * Writes data for the current entry to the underlying stream.
     *
     * @throws IOException
     *                If an error occurs writing to the stream
     */
    @Override
    public void write(byte[] buffer, int offset, int byteCount) throws IOException {
        Arrays.checkOffsetAndCount(buffer.length, offset, byteCount);
        if (currentEntry == null) {
            throw new ZipException("No active entry");
        }

        final long totalBytes = crc.tbytes + byteCount;
        if ((totalBytes > Zip64.MAX_ZIP_ENTRY_AND_ARCHIVE_SIZE) && !currentEntryNeedsZip64) {
            throw new IOException("Zip entry size (" + totalBytes +
                    " bytes) cannot be represented in the zip format (needs Zip64)." +
                    " Set the entry length using ZipEntry#setLength to use Zip64 where necessary.");
        }

        if (currentEntry.getMethod() == STORED) {
            out.write(buffer, offset, byteCount);
        } else {
            super.write(buffer, offset, byteCount);
        }
        crc.update(buffer, offset, byteCount);
    }

    private void checkOpen() throws IOException {
        if (cDir == null) {
            throw new IOException("Stream is closed");
        }
    }

    private void checkSizeIsWithinShort(String property, byte[] bytes) {
        if (bytes.length > 0xffff) {
            throw new IllegalArgumentException(property + " too long in UTF-8:" + bytes.length +
                                               " bytes");
        }
    }

    private long getBytesWritten() {
        // This cast is somewhat messy but less error prone than keeping an
        // CountingOutputStream reference around in addition to the FilterOutputStream's
        // out.
        return ((CountingOutputStream) out).getCount();
    }
}


text:
package org.elasticsearch.xpack.ml;

import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.metadata.IndexAbstraction;
import org.elasticsearch.cluster.routing.IndexRoutingTable;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.settings.Setting;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.xpack.core.ml.MlConfigIndex;

/**
 * Checks whether ML config migration can start.
 *
 * Originally this class was used to check whether jobs and datafeeds could be
 * migrated from cluster state to the ML config index. This use case is no longer
 * relevant. However, it's possible that in the future we may want to do some
 * other form of config migration, and the cluster setting defined in this class
 * would be an ideal way to restrict it.
 */
public class MlConfigMigrationEligibilityCheck {

    public static final Setting<Boolean> ENABLE_CONFIG_MIGRATION = Setting.boolSetting(
        "xpack.ml.enable_config_migration",
        true,
        Setting.Property.OperatorDynamic,
        Setting.Property.NodeScope
    );

    private volatile boolean isConfigMigrationEnabled;

    public MlConfigMigrationEligibilityCheck(Settings settings, ClusterService clusterService) {
        isConfigMigrationEnabled = ENABLE_CONFIG_MIGRATION.get(settings);
        clusterService.getClusterSettings().addSettingsUpdateConsumer(ENABLE_CONFIG_MIGRATION, this::setConfigMigrationEnabled);
    }

    private void setConfigMigrationEnabled(boolean configMigrationEnabled) {
        this.isConfigMigrationEnabled = configMigrationEnabled;
    }

    /**
     * Can migration start? Returns:
     *     False if config migration is disabled via the setting {@link #ENABLE_CONFIG_MIGRATION}
     *     False if the .ml-config index shards are not active
     *     True otherwise
     * @param clusterState The cluster state
     * @return A boolean that dictates if config migration can start
     */
    public boolean canStartMigration(ClusterState clusterState) {
        if (isConfigMigrationEnabled == false) {
            return false;
        }
        return mlConfigIndexIsAllocated(clusterState);
    }

    static boolean mlConfigIndexIsAllocated(ClusterState clusterState) {
        IndexAbstraction configIndexOrAlias = clusterState.metadata().getIndicesLookup().get(MlConfigIndex.indexName());
        if (configIndexOrAlias == null) {
            return false;
        }

        IndexRoutingTable routingTable = clusterState.getRoutingTable().index(configIndexOrAlias.getWriteIndex());
        return routingTable != null && routingTable.allPrimaryShardsActive();
    }
}


text:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/batch"
             xmlns:beans="http://www.springframework.org/schema/beans"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://www.springframework.org/schema/beans
			http://www.springframework.org/schema/beans/spring-beans.xsd">

    <beans:bean name="jacksonMapper"
                class="com.fasterxml.jackson.databind.ObjectMapper" />

</beans:beans>


text:
<!DOCTYPE html>

        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Cleaning | Greentheme Services</title>
        <meta name="description" content="Cleaning | Greentheme Services">
        <meta name="keywords" content="greentheme services, cleaning">
        <meta name="author" content="Peter Champion">

        <link rel="stylesheet" href="css/normalize.css">
        <link rel="stylesheet" href="css/main.css">
        <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css" rel="stylesheet">

        <link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>

Home

FAQ

Blog

Services

lawns

household handyman

cleaning

garden maintenance

weedspraying

shop signs and windows

carpet cleaning

housewash

Contact

lawns

Greentheme Lawn Services offers for the home owner-with attention to quality-edges done routinely with a line trimmer.For the rental property-with attention to price-basically an affordable option to the landlord or tenant doing it themselves. Premium Lawns using a seed free mower for the premium lawn.

household handyman

Greentheme can do handymans jobs around your house,with a welcome difference! We are highly skilled for each job. Just have a look at our skill set such as building fences, repairing fences, building repairs, small construction. In fact, any building jobs, all undertaken by qualified, registered builders.

cleaning

Well you're gonna get clean.
                        Greentheme Services have cleaners in both the domestic and commercial cleaning sectors.

garden maintenance

Not only can Greentheme Services tame your wild garden they can design a new one! We have an experienced garden designer in-house who can discuss your garden and how you would like it to be for your ultimate enjoyment.

weedspraying

Are you sick of looking at weeds poking out from under your hedge where the mower can't get to it or invading your driveway through the cracks? Well Greentheme Services can fix that very easily! An average house only costs around $65 plus GST for a complete maintenance spray. Also if you want to upgrade your lawn and replant we can spray the existing lawn and you can do the rest.

shop signs and windows

Have you ever wandered down the street as a potential customer or shopper and seen how grubby the shop and business signs are under the awnings and on the front of buildings? If you are a business owner think about it, it's probably the first thing they are looking for when they are visiting your premises. Road film is quite often the culprit and mix that with dust and other contaminates and you have the recipe for a hard to remove sludge.

carpet cleaning

We offer the best hi tech equipment is used for a top quality clean. Depending on your property and carpet type we use a hot water rotary jet extraction or a straight extraction method which draws the dirt\water mix from your carpet leaving it super clean with a minimal drying time.

housewash

Because we have the equipment to wash signage it is a simple transaction to wash houses as well as the ability to wash buildings. We have commercial detergents that will remove dirt and mould and algae.

                    mobile ready, touch swipe

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

                    finger catchable right to left

            javascript

So you want Clean huh!

Well you're gonna get clean.

Greentheme Services have cleaners in both the domestic and commercial cleaning sectors.

House Cleaning

Greentheme Services have a team of cleaners that are not happy with anything that is less than perfect.

Call Greentheme Services if you are moving out of a rented house or if you are a landlord wanting a rental property clean for the next tenant.

Also, if you have an elderly relative who wants regular cleaning in their house, call us no problem!

Ovens and Fridges

Greentheme can arrive at your house and clean either your oven or your fridge (or both!)

We will also clean your kitchen or restroom only it doesn't have to be the whole house

Commercial Cleaning

Greentheme Services have experienced cleaners in the commercial sector who are also proud of their quality of work.

In any of our cleaning services we listen to our clients regarding their special items of interest, they are all different, and we pay particular attention to that area.

Developed by: Jamal Fortes


text:
'''
Created on Aug 26, 2012

@author: Mark V Systems Limited
(c) Copyright 2012 Mark V Systems Limited, All rights reserved.
'''
import os, time, io, ast, sys, traceback

def entityEncode(arg):  # be sure it's a string, vs int, etc, and encode &, <, ".
    return str(arg).replace("&","&").replace("<","<").replace('"','"')

if __name__ == "__main__":
    startedAt = time.time()

    idMsg = []
    numArelleSrcFiles = 0

    arelleSrcPath = (os.path.dirname(__file__) or os.curdir) + os.sep + "arelle"
    for arelleSrcDir in (arelleSrcPath,
                         arelleSrcPath + os.sep + "plugin",
                         arelleSrcPath + os.sep + "plugin" + os.sep + "validate" + os.sep + "EFM",
                         arelleSrcPath + os.sep + "plugin" + os.sep + "EdgarRenderer"
                         # arelleSrcPath + os.sep + "plugin" + os.sep + "validate" + os.sep + "GL"
                         ):
        for moduleFilename in os.listdir(arelleSrcDir):
            if moduleFilename.endswith(".py"):
                numArelleSrcFiles += 1
                fullFilenamePath = arelleSrcDir + os.sep + moduleFilename
                refFilename = fullFilenamePath[len(arelleSrcPath)+1:].replace("\\","/")
                with open(fullFilenamePath, encoding="utf-8") as f:
                    tree = ast.parse(f.read(), filename=moduleFilename)
                    for item in ast.walk(tree):
                        try:
                            if (isinstance(item, ast.Call) and
                                (getattr(item.func, "attr", '') or getattr(item.func, "id", '')) # imported function could be by id instead of attr
                                in ("info","warning","log","error","exception")):
                                    funcName = item.func.attr
                                    iArgOffset = 0
                                    if funcName == "info":
                                        level = "info"
                                    elif funcName == "warning":
                                        level = "warning"
                                    elif funcName == "error":
                                        level = "error"
                                    elif funcName == "exception":
                                        level = "exception"
                                    elif funcName == "log":
                                        levelArg = item.args[0]
                                        if isinstance(levelArg,ast.Str):
                                            level = levelArg.s.lower()
                                        else:
                                            if any(isinstance(elt, (ast.Call, ast.Name))
                                                   for elt in ast.walk(levelArg)):
                                                level = "(dynamic)"
                                            else:
                                                level = ', '.join(elt.s.lower()
                                                                  for elt in ast.walk(levelArg)
                                                                  if isinstance(elt, ast.Str))
                                        iArgOffset = 1
                                    msgCodeArg = item.args[0 + iArgOffset]  # str or tuple
                                    if isinstance(msgCodeArg,ast.Str):
                                        msgCodes = (msgCodeArg.s,)
                                    elif isinstance(msgCodeArg, ast.Call) and getattr(msgCodeArg.func, "id", '') == 'ixMsgCode':
                                        msgCodes = ("ix{{ver.sect}}:{}".format(msgCodeArg.args[0].s),)
                                    else:
                                        if any(isinstance(elt, (ast.Call, ast.Name))
                                               for elt in ast.walk(msgCodeArg)):
                                            msgCodes = ("(dynamic)",)
                                        else:
                                            msgCodes = [elt.s
                                                        for elt in ast.walk(msgCodeArg)
                                                        if isinstance(elt, ast.Str)]
                                    msgArg = item.args[1 + iArgOffset]
                                    if isinstance(msgArg, ast.Str):
                                        msg = msgArg.s
                                    elif isinstance(msgArg, ast.Call) and getattr(msgArg.func, "id", '') == '_':
                                        msg = msgArg.args[0].s
                                    elif any(isinstance(elt, (ast.Call,ast.Name))
                                             for elt in ast.walk(msgArg)):
                                        msg = "(dynamic)"
                                    else:
                                        continue # not sure what to report
                                    keywords = []
                                    for keyword in item.keywords:
                                        if keyword.arg == 'modelObject':
                                            pass
                                        elif keyword.arg == 'messageCodes':
                                            msgCodeArg = keyword.value
                                            if any(isinstance(elt, (ast.Call, ast.Name))
                                                   for elt in ast.walk(msgCodeArg)):
                                                pass # dynamic
                                            else:
                                                msgCodes = [elt.s
                                                            for elt in ast.walk(msgCodeArg)
                                                            if isinstance(elt, ast.Str)]
                                        else:
                                            keywords.append(keyword.arg)
                                    for msgCode in msgCodes:
                                        idMsg.append((msgCode, msg, level, keywords, refFilename, item.lineno))
                        except (AttributeError, IndexError):
                            pass

    lines = []
    for id,msg,level,args,module,line in idMsg:
        try:
            if args and any(isinstance(arg,str) for arg in args):
                argAttr = "\n         args=\"{0}\"".format(
                            entityEncode(" ".join(arg for arg in args if arg is not None)))
            else:
                argAttr = ""
            lines.append("<message code=\"{0}\"\n         level=\"{3}\"\n         module=\"{4}\" line=\"{5}\"{2}>\n{1}\n</message>"
                      .format(id,
                              entityEncode(msg),
                              argAttr,
                              level,
                              module,
                              line))
        except Exception as ex:
            print(ex)
            print("traceback {}".format(traceback.format_tb(sys.exc_info()[2])))
    os.makedirs(arelleSrcPath + os.sep + "doc", exist_ok=True)
    with io.open(arelleSrcPath + os.sep + "doc" + os.sep + "messagesCatalog.xml", 'wt', encoding='utf-8') as f:
        f.write(
'''<?xml version="1.0" encoding="utf-8"?>
<messages
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="messagesCatalog.xsd"
    variablePrefix="%("
    variableSuffix=")s"
    variablePrefixEscape="" >
<!--
This file contains Arelle messages text.   Each message has a code
that corresponds to the message code in the log file, level (severity),
args (available through log file), and message replacement text.

(Messages with dynamically composed error codes or text content
(such as ValidateXbrlDTS.py line 158 or lxml parser messages)
are reported as "(dynamic)".)

-->

''')
        f.write("\n\n".join(sorted(lines)))
        f.write("\n\n</messages>")

    with io.open(arelleSrcPath + os.sep + "doc" + os.sep + "messagesCatalog.xsd", 'wt', encoding='utf-8') as f:
        f.write(
'''<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="unqualified"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <xs:element name="messages">
    <xs:complexType>
      <xs:sequence>
        <xs:element maxOccurs="unbounded" name="message">
          <xs:complexType>
            <xs:simpleContent>
              <xs:extension base="xs:string">
                <xs:attribute name="code" use="required" type="xs:normalizedString"/>
                <xs:attribute name="level" use="required" type="xs:token"/>
                <xs:attribute name="module" type="xs:normalizedString"/>
                <xs:attribute name="line" type="xs:integer"/>
                <xs:attribute name="args" type="xs:NMTOKENS"/>
              </xs:extension>
            </xs:simpleContent>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
      <xs:attribute name="variablePrefix" type="xs:string"/>
      <xs:attribute name="variableSuffix" type="xs:string"/>
      <xs:attribute name="variablePrefixEscape" type="xs:string"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
''')

    print("Arelle messages catalog {0:.2f} secs, {1} formula files, {2} messages".format( time.time() - startedAt, numArelleSrcFiles, len(idMsg) ))