text:
"""
Parameter expression evaluator.

For systems in which constraints are expressed as string expressions rather
than python code, :func:`compile_constraints` can construct an expression
evaluator that substitutes the computed values of the expressions into the
parameters.

The compiler requires a symbol table, an expression set and a context.
The symbol table maps strings containing fully qualified names such as
'M1.c[3].full_width' to parameter objects with a 'value' property that
can be queried and set.  The expression set maps symbol names from the
symbol table to string expressions.  The context provides additional symbols
for the expressions in addition to the usual mathematical functions and
constants.

The expressions are compiled and interpreted by python, with only minimal
effort to make sure that they don't contain bad code.  The resulting
constraints function returns 0 so it can be used directly in a fit problem
definition.

Extracting the symbol table from the model depends on the structure of the
model.  If fitness.parameters() is set correctly, then this should simply
be a matter of walking the parameter data, remembering the path to each
parameter in the symbol table.  For compactness, dictionary elements should
be referenced by .name rather than ["name"].  Model name can be used as the
top level.

Getting the parameter expressions applied correctly is challenging.
The following monkey patch works by overriding model_update in FitProblem
so that after setp(p) is called and, the constraints expression can be
applied before telling the underlying fitness function that the model
is out of date::

        # Override model update so that parameter constraints are applied
        problem._model_update = problem.model_update
        def model_update():
            constraints()
            problem._model_update()
        problem.model_update = model_update

Ideally, this interface will change
"""
from __future__ import print_function

from copy import copy
import math
import re
from keyword import iskeyword

def standard_symbols(context={}):
    symbols = {}
    symbols.update(math.__dict__)
    symbols.update(dict(arcsin=math.asin, arccos=math.acos,
                        arctan=math.atan, arctan2=math.atan2))
    symbols.update(context)
    symbols['id'] = id
    return symbols

def _check_syntax(target, expr, html=False):
    try:
        compile(expr, expr, "exec")
    except SyntaxError as exc:
        if html:
            if "\n" in expr:
                # Multiline expression. Be lazy and just show line, col for
                # syntax error since this should never happen.
                return [
                    f"Syntax error on line {exc.lineno} column {exc.offset} for {target}:\n
\n{expr}\n
"]

            if exc.offset > len(expr):
                # Single line expression with error after the expression.
                # Probably missing a closing paren, but it could
                # also be that the expression ends with an operator such as
                # "3+4+"
                return [f"Expression `{target} = {expr}` is not complete."]

            # Single line expression. Wrap the tail of the expr after the
            # syntax error in .... exc.lineno=1, exc.text = expr+"\n",
            # and exc.offset = location of the syntax error in expr.
            return [
                f"Syntax error in expression '{target} = {expr[:exc.offset - 1]}{expr[exc.offset - 1:]}'"]
        else:
            return ["Syntax error in expression '%s = %s'" % (target, expr)]
    return []

def _check_free_variables(target, expr, symbol_table, html=False):
    undefined = [sym for sym in _symbols(expr)
                 if sym not in symbol_table and not iskeyword(sym)]
    if undefined:
        undefined_str = ", ".join(sorted(undefined))
        if html:
            for symbol in undefined:
                # Identify the symbol for replacement as everything between a
                # word boundary. Since symbols can contain '.', we need to
                # use negative lookbehind and negative lookahead on a
                # character set containing '.' to check for the boundaries.
                # Also, if there is a '.' in the symbol it could match any
                # character, so it needs to be turned into a regular
                # expression that matches '.'.
                pattern = f"(?<![a-zA-Z0-9_.]){symbol.replace('.', '[.]')}(?![a-zA-Z0-9_.])"
                expr = re.sub(pattern, f"{symbol}", expr)
            return ["Unknown parameters (%s) in expression '%s = %s'"
                    % (undefined_str, target, expr)]
        else:
            return ["Unknown parameters (%s) in expression '%s = %s'"
                    % (undefined_str, target, expr)]
    return []

# simple pattern which matches symbols.  Note that it will also match
# invalid substrings such as a3...9, but given syntactically correct
# input it will only match symbols.
_SYMBOL_PATTERN = re.compile('([a-zA-Z_][a-zA-Z_0-9.]*)')
def _symbols(expr):
    return set(m.group(0) for m in _SYMBOL_PATTERN.finditer(expr))

def _substitute(expr, mapping):
    """
    Replace all occurrences of symbol s with mapping[s] for s in mapping.
    """
    # Find the symbols and the mapping
    matches = [(m.start(), m.end(), mapping[m.group(1)])
               for m in _SYMBOL_PATTERN.finditer(expr)
               if m.group(1) in mapping]

    # Split the expression in to pieces, with new symbols replacing old
    pieces = []
    offset = 0
    for start, end, text in matches:
        pieces += [expr[offset:start], text]
        offset = end
    pieces.append(expr[offset:])

    # Join the pieces and return them
    return "".join(pieces)

def _find_dependencies(symtab, exprs):
    """
    Returns a list of pair-wise dependencies from the parameter expressions.

    *symtab* gives the *{'parameter': value}* table of available parameters.

    *exprs* gives the *{'parameter': 'expr'}* table of parameter expressions.

    For example, if p3 = p1+p2, then find_dependencies([p1, p2, p3]) will
    return *[('p3', 'p1'), ('p3', 'p2')]*. For base expressions without
    dependencies, such as p4 = 2*pi, this should return *[('p4', None)]*.
    """
    deps = [(target, source)
            for target, expr in exprs.items()
            for source in _dependent_symbols(expr, symtab)]
    return deps

# Hack to deal with expressions without dependencies --- return a fake
# dependency of None.
# The better solution is fix order_dependencies so that it takes a
# dictionary of {symbol: dependency_list}, for which no dependencies
# is simply []; fix in parameter_mapping as well
def _dependent_symbols(expr, symtab):
    """
    Given an expression string and a symbol table, return the set of symbols
    used in the expression. Symbols are only returned once even if they occur
    multiple times. The return value is a set with the elements in no
    particular order. Returns a set containing *None* if no dependencies (as
    needed by *order_dependencies*).

    This is the first step in computing a dependency graph.
    """
    deps = set([m for m in _symbols(expr) if m in symtab])
    return deps if deps else {None}

def _parameter_mapping(pairs):
    """
    Find the parameter substitution we need so that expressions can
    be evaluated without having to traverse a chain of
    model.layer.parameter.value
    """
    left, right = zip(*pairs)
    pars = list(sorted(p for p in set(left+right) if p is not None))
    definition = dict( ('P%d'%i, p)  for i, p in enumerate(pars) )
    # p is None when there is an expression with no dependencies
    substitution = dict((p, 'P%d.value'%i)
                        for i, p in enumerate(sorted(pars))
                        if p is not None)
    return definition, substitution

def no_constraints():
    """
    This parameter set has no constraints between the parameters.
    """
    pass

def compile_constraints(symtab, exprs, context={}):
    """
    Build and return a function to evaluate all parameter expressions in
    the proper order.

    Input:

        *symtab* is the symbol table for the model: { 'name': parameter }

        *exprs* is the set of computed symbols: { 'name': 'expression' }

        *context* is any additional context needed to evaluate the expression

    Return:

        updater function which sets parameter.value for each expression

    Raises:

       RunTimeError if any expression contains a syntax error, if any
       symbol used is not defined, or if there are circular dependencies
       between symbols. Runtime error argument is a string describing all
       found errors.

    This function is not terribly sophisticated, and it would be easy to
    trick.  However it handles the common cases cleanly and generates
    reasonable messages for the common errors.

    This code has not been fully audited for security.  While we have
    removed the builtins and the ability to import modules, there may
    be other vectors for users to perform more than simple function
    evaluations. Unauthenticated users should not be running this code.

    Parameter names are assumed to contain only _.a-zA-Z0-9#[]

    Both names are provided for inverse functions, e.g., acos and arccos.

    Should try running the function to identify syntax errors before
    running it in a fit.

    Use help(fn) to see the code generated for the returned function fn.
    dis.dis(fn) will show the corresponding python vm instructions.
    """
    retfn, errors = _compile_constraints(symtab, exprs, context=context)
    if errors:
        raise RuntimeError("\n".join(errors))
    return retfn

# Simple parameter class for checking constraints
class _Parameter:
    def __init__(self, value=0):
        self.value = value

def check_constraints(symtab, exprs, context={}, html=False):
    """
    Returns a list of errors in *exprs* or the empty list if there are none.
    If the html flag is set to True, the list elements will have html
    markups that allow the caller to control rendering:

    Unknown symbol: tags unknown symbols in *exprs*
    Syntax error: tags the beginning of a syntax error in *exprs*
    Cyclic dependency: tags comma separated parameters that have
    cyclic dependency

    All symbols must exist in *context* or in *symtab*. The symbols in
    *context* should be constants or functions. The symbols in *symtab*
    can be constants or parameter objects with a *value* attribute.

    It first runs :func:`compile_constraints`, returning a list of errors
    if any. If there are no errors it runs the compiled constraints function,
    returning any errors it produces. Any parameters in *symtab* are copied
    with a shallow copy so they aren't overridden when the constraints are run.
    """
    # Make sure the symbols are wrapped in parameter objects. This allows
    # us to run the function and check constraints.
    symtab = {k: (copy(v) if hasattr(v, 'value') else _Parameter(v))
              for k, v in symtab.items()}
    retfn, errors = _compile_constraints(symtab,
                                         exprs,
                                         context=context,
                                         html=html)
    if not errors:
        try:
            retfn()
        except Exception as exc:
            errors.append(str(exc))
    return errors

def _compile_constraints(symtab, exprs, context={}, html=False):
    errors = []

    # Check the syntax before compiling the complete function.
    available_symbols = standard_symbols(context)
    available_symbols.update(symtab)
    for k, v in exprs.items():
        errors.extend(_check_syntax(k, v, html=html))
        errors.extend(_check_free_variables(k, v, available_symbols, html=html))

    # Sort the parameters in the order they need to be evaluated
    # Note: order_dependencies raises an error if there are cyclic dependencies
    deps = _find_dependencies(symtab, exprs)
    if not deps:
        return no_constraints, []
    try:
        order = order_dependencies(deps)
    except Exception as exc:
        if html:
            errors.append("Cyclic dependency amongst parameters: "
                          "%s"
                          % str(exc))
        else:
            errors.append("Cyclic dependency amongst parameters: %s" % str(exc))

    if errors:
        return None, errors
    #print(f"{symtab=}\n  {deps=}\n  {order=}\n")

    # Rather than using the full path to the parameters in the parameter
    # expressions, instead use Pn, and substitute Pn.value for each occurrence
    # of the parameter in the expression.
    names = list(sorted(symtab.keys()))
    parameters = dict(('P%d'%i, symtab[k]) for i, k in enumerate(names))
    mapping = dict((k, 'P%d.value'%i) for i, k in enumerate(names))

    # Add the parameters to the global context
    global_context = standard_symbols(context)
    global_context.update(parameters)
    local_context = {}

    # Define the constraints function
    assignments = ["=".join((p, exprs[p])) for p in order]
    code = [_substitute(s, mapping) for s in assignments]
    # TODO: maybe wrap body in try...except block and return NaN?
    functiondef = """
def eval_expressions():
    '''
    %s
    '''
    %s
    return 0
"""%("\n    ".join(assignments), "\n    ".join(code))

    #print(" ", "\n  ".join(code))
    #print("Function: "+functiondef)
    # CRUFT: python < 3.0;  doc builder isn't allowing the following exec
    # https://stackoverflow.com/questions/4484872/why-doesnt-exec-work-in-a-function-with-a-subfunction/41368813#comment73790496_41368813
    #exec(functiondef, global_context, local_context)
    source = functiondef
    location = "\n  ".join(assignments)
    eval(compile(source, location, 'exec'), global_context, local_context)
    retfn = local_context['eval_expressions']

    # Remove garbage added to globals by exec
    global_context.pop('__doc__', None)
    global_context.pop('__name__', None)
    global_context.pop('__file__', None)
    global_context.pop('__builtins__')
    #print globals.keys()

    return retfn, errors

def order_dependencies(pairs):
    """
    Order elements from pairs so that b comes before a in the
    ordered list for all pairs (a, b).
    """
    emptyset = set()
    order = []

    # Break pairs into left set and right set
    # Note: pairs is array or list, so use "len(pairs) > 0" to check for empty.
    left, right = [set(s) for s in zip(*pairs)] if len(pairs) > 0 else ([], [])
    while len(pairs) > 0:
        # Find which items only occur on the right
        independent = right - left
        if independent == emptyset:
            cycleset = ", ".join(str(s) for s in left)
            raise ValueError(cycleset)

        # The possibly resolvable items are those that depend on the independents
        dependent = set([a for a, b in pairs if b in independent])
        pairs = [(a, b) for a, b in pairs if b not in independent]
        if len(pairs) == 0:
            resolved = dependent
        else:
            left, right = [set(s) for s in zip(*pairs)]
            resolved = dependent - left

        order += resolved

    return order

# ========= Test code ========
def _check(msg, pairs):
    """
    Verify that the list n contains the given items, and that the list
    satisfies the partial ordering given by the pairs in partial order.
    """
    # pairs are a list of (lhs, rhs)
    # lhs may be repeated e.g., x = a+b has pairs (x, a) and (x, b)
    # lhs may be in rhs e.g., x = a+b; b = 2*c has pairs (x, a) (x, b), (b, c)
    # find lhs eval order; since (x, b) is a pair then b must come before x
    # Note: pairs is array or list, so use "len(pairs) > 0" to check for empty.
    lhs_list, rhs_list = zip(*pairs) if len(pairs) > 0 else ([], [])
    items = set(lhs_list) # items = all LHS, removing duplicates
    order = order_dependencies(pairs)
    if set(order) != set(items) or len(order) != len(items):
        raise ValueError("%s expect %s to contain %s for %s"
                         % (msg, order, sorted(items), pairs))
    for lhs, rhs in pairs:
        if lhs in order and rhs in order and order.index(rhs) >= order.index(lhs):
            raise ValueError("%s expect %s before %s in %s for %s"
                             % (msg, lhs, rhs, order, pairs))

def test_deps():
    import numpy as np

    # Null case
    _check("test empty", [])

    # Some dependencies
    _check("test1", [(2, 7), (1, 5), (1, 4), (2, 1), (3, 1), (5, 6)])
    _check("test1 renumbered",
           [(6, 1), (7, 3), (7, 4), (6, 7), (5, 7), (3, 2)])
    _check("test1 numpy",
           np.array([(2, 7), (1, 5), (1, 4), (2, 1), (3, 1), (5, 6)]))

    # No dependencies
    _check("test2", [(4, 1), (3, 2), (8, 4)])

    # Cycle test
    pairs = [(1, 4), (4, 3), (4, 5), (5, 1)]
    try:
        n = order_dependencies(pairs)
    except ValueError:
        pass
    else:
        raise ValueError("test3 expect ValueError exception for %s" % (pairs,))

    # large test for gross speed check
    A = np.random.randint(4000, size=(1000, 2))
    A[:, 1] += 4000  # Avoid cycles
    _check("test-large", A)

    # depth tests
    k = 200
    A = np.array([range(0, k), range(1, k+1)]).T
    _check("depth-1", A)

    A = np.array([range(1, k+1), range(0, k)]).T
    _check("depth-2", A)

def test_expr():
    import inspect
    import dis

    symtab = {'a.b.x': 1, 'a.c': 2, 'a.b': 3, 'b.x': 4}
    expr = 'a.b.x + sin(4*pi*a.c) + a.b.x/a.b'

    # Check symbol lookup
    assert _dependent_symbols(expr, symtab) == set(['a.b.x', 'a.c', 'a.b'])

    # Check symbol rename
    assert _substitute(expr, {'a.b.x': 'Q'}) == 'Q + sin(4*pi*a.c) + Q/a.b'
    assert _substitute(expr, {'a.b': 'Q'}) == 'a.b.x + sin(4*pi*a.c) + a.b.x/Q'

    # Check dependency builder
    # Fake parameter class
    class TestParameter:
        def __init__(self, name, value=0, expression=''):
            self.path = name
            self.value = value
            self.expression = expression
        def iscomputed(self):
            return (self.expression != '')
        def __repr__(self):
            value = self.expression if self.iscomputed() else str(self.value)
            return self.path + '=' + value
    def world(*pars):
        symtab = dict((p.path, p) for p in pars)
        exprs = dict((p.path, p.expression) for p in pars if p.iscomputed())
        return symtab, exprs
    p1 = TestParameter('G0.sigma', 5)
    p2 = TestParameter('other', expression='2*pi*sin(G0.sigma/.1875) + M1.G1')
    p3 = TestParameter('M1.G1', 6)
    p3_circular = TestParameter('M1.G1', expression='other + 6')
    p3_self = TestParameter('M1.G1', expression='M1.G1')
    p4 = TestParameter('constant', expression='2*pi*35')
    p5 = TestParameter('chain', expression='2+other')
    # Simple pairs
    assert (set(_find_dependencies(*world(p1, p2, p3)))
            == set([(p2.path, p1.path), (p2.path, p3.path)]))
    # Chain
    assert (set(_find_dependencies(*world(p1, p2, p3, p5)))
            == set([(p2.path, p1.path), (p2.path, p3.path), (p5.path, p2.path)]))
    # Constant expression
    assert set(_find_dependencies(*world(p1, p4))) == set([(p4.path, None)])
    # No dependencies
    assert not set(_find_dependencies(*world(p1, p3)))

    # Make sure 'other' is evaluated before 'chain'
    assert (order_dependencies(_find_dependencies(*world(p1, p2, p3, p5)))
            == [p2.path, p5.path])
    # Check function builder
    fn = compile_constraints(*world(p1, p2, p3))

    # Inspect the resulting function
    if 0:
        print(inspect.getdoc(fn))
        print(dis.dis(fn))

    # Evaluate the function and see if it updates the
    # target value as expected
    fn()
    expected = 2*math.pi*math.sin(5/.1875) + 6
    assert p2.value == expected, "Value was %s, not %s" % (p2.value, expected)

    # Make sure check_constraints returns an empty list for these expressions.
    assert not check_constraints(*world(p1, p2, p3))

    # Check empty dependency set doesn't crash
    fn = compile_constraints(*world(p1, p3))
    fn()

    # Check that constants are evaluated properly
    fn = compile_constraints(*world(p4))
    fn()
    assert p4.value == 2*math.pi*35

    # Check that circular definitions get flagged
    try:
        fn = compile_constraints(*world(p1, p2, p3_circular))
    except Exception as exc:
        assert str(exc).startswith('Cyclic')
    else:
        raise RuntimeError("failed to raise error for cyclic dependency")
    try:
        fn = compile_constraints(*world(p1, p2, p3_self))
    except Exception as exc:
        assert str(exc).startswith('Cyclic')
    else:
        raise RuntimeError("failed to raise error for self dependency")

    # Make sure errors are returned from check_constraints
    errors = check_constraints(*world(p1, p2, p3_circular))
    assert len(errors) == 1 and errors[0].startswith('Cyclic')
    errors = check_constraints(*world(p1, p2, p3_self))
    assert len(errors) == 1 and errors[0].startswith('Cyclic')

    # Check additional context example; this also tests multiple expressions
    tbl = {'tbl_Si': 2.09}
    p5 = TestParameter('lookup', expression="tbl_Si")
    fn = compile_constraints(*world(p1, p2, p3, p5), context=tbl)
    fn()
    assert p5.value == 2.09, "Value for %s was %s" % (p5.expression, p5.value)

    #class Table:
    #    Si = 2.09
    #    values = {'Si': 2.07}
    #tbl = Table()
    #p5 = TestParameter('lookup', expression="tbl.Si")
    #fn = compile_constraints(*world(p1, p2, p3, p5), context=dict(tbl=tbl))
    #fn()
    #assert p5.value == 2.09, "Value for %s was %s"%(p5.expression, p5.value)
    #
    #p5.expression = "tbl.values['Si']"
    #fn = compile_constraints(*world(p1, p2, p3, p5), context=dict(tbl=tbl))
    #fn()
    #assert p5.value == 2.07, "Value for %s was %s" % (p5.expression, p5.value)

    # Verify that we capture invalid expressions
    for expr in ['G4.cage', 'M0.cage', 'M1.G1 + *2',
                 'piddle',
                 #'5; import sys; print("p0wned")',
                 '__import__("sys").argv'
                 ]:
        try:
            p6 = TestParameter('broken', expression=expr)
            fn = compile_constraints(*world(p6))
            fn()
        except Exception as msg:
            #print(msg)
            pass
        else:
            raise RuntimeError("Failed to raise error for %s" % (expr,))

    # Verify that check_constraints returns multiple errors.
    symtab = {
        'M1.sld': 1.0,
        'M1.sld_solvent': 6.0,
        'M1.radius': 50,
        'M1.scale': 1.0,
    }
    exprs = {
        'M1.background': 'M1.scal/1e5',
        'M1.sld': 'M1.sld_solvent + 1',
        'M1.sld_solvent': 'M1.sld + 2',
    }
    errors = check_constraints(symtab, exprs)
    assert len(errors) == 2
    #print("\n".join(errors))

if __name__ == "__main__":
    test_expr()
    test_deps()


text:
package org.apache.cassandra.transport;

import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import java.nio.charset.CharacterCodingException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;

import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.util.CharsetUtil;

import org.apache.cassandra.db.ConsistencyLevel;
import org.apache.cassandra.utils.UUIDGen;

/**
 * ChannelBuffer utility methods.
 * Note that contrarily to ByteBufferUtil, these method do "read" the
 * ChannelBuffer advancing it's (read) position. They also write by
 * advancing the write position. Functions are also provided to create
 * ChannelBuffer while avoiding copies.
 */
public abstract class CBUtil
{
    private CBUtil() {}

    public static String readString(ChannelBuffer cb)
    {
        try
        {
            int length = cb.readUnsignedShort();
            return readString(cb, length);
        }
        catch (IndexOutOfBoundsException e)
        {
            throw new ProtocolException("Not enough bytes to read an UTF8 serialized string preceded by it's 2 bytes length");
        }
    }

    public static String readLongString(ChannelBuffer cb)
    {
        try
        {
            int length = cb.readInt();
            return readString(cb, length);
        }
        catch (IndexOutOfBoundsException e)
        {
            throw new ProtocolException("Not enough bytes to read an UTF8 serialized string preceded by it's 4 bytes length");
        }
    }

    private static String readString(ChannelBuffer cb, int length)
    {
        try
        {
            String str = cb.toString(cb.readerIndex(), length, CharsetUtil.UTF_8);
            cb.readerIndex(cb.readerIndex() + length);
            return str;
        }
        catch (IllegalStateException e)
        {
            // That's the way netty encapsulate a CCE
            if (e.getCause() instanceof CharacterCodingException)
                throw new ProtocolException("Cannot decode string as UTF8");
            else
                throw e;
        }
    }

    private static ChannelBuffer bytes(String str)
    {
        return ChannelBuffers.wrappedBuffer(str.getBytes(CharsetUtil.UTF_8));
    }

    public static ChannelBuffer shortToCB(int s)
    {
        ChannelBuffer cb = ChannelBuffers.buffer(2);
        cb.writeShort(s);
        return cb;
    }

    public static ChannelBuffer intToCB(int i)
    {
        ChannelBuffer cb = ChannelBuffers.buffer(4);
        cb.writeInt(i);
        return cb;
    }

    public static ChannelBuffer stringToCB(String str)
    {
        ChannelBuffer bytes = bytes(str);
        return ChannelBuffers.wrappedBuffer(shortToCB(bytes.readableBytes()), bytes);
    }

    public static ChannelBuffer bytesToCB(byte[] bytes)
    {
        return ChannelBuffers.wrappedBuffer(shortToCB(bytes.length), ChannelBuffers.wrappedBuffer(bytes));
    }

    public static byte[] readBytes(ChannelBuffer cb)
    {
        try
        {
            int length = cb.readUnsignedShort();
            byte[] bytes = new byte[length];
            cb.readBytes(bytes);
            return bytes;
        }
        catch (IndexOutOfBoundsException e)
        {
            throw new ProtocolException("Not enough bytes to read a byte array preceded by it's 2 bytes length");
        }
    }

    public static ChannelBuffer consistencyLevelToCB(ConsistencyLevel consistency)
    {
        return shortToCB(consistency.code);
    }

    public static ConsistencyLevel readConsistencyLevel(ChannelBuffer cb)
    {
        return ConsistencyLevel.fromCode(cb.readUnsignedShort());
    }

    public static <T extends Enum<T>> T readEnumValue(Class<T> enumType, ChannelBuffer cb)
    {
        String value = CBUtil.readString(cb);
        try
        {
            return Enum.valueOf(enumType, value.toUpperCase());
        }
        catch (IllegalArgumentException e)
        {
            throw new ProtocolException(String.format("Invalid value '%s' for %s", value, enumType.getSimpleName()));
        }
    }

    public static <T extends Enum<T>> ChannelBuffer enumValueToCB(T enumValue)
    {
        return stringToCB(enumValue.toString());
    }

    public static ChannelBuffer uuidToCB(UUID uuid)
    {
        return ChannelBuffers.wrappedBuffer(UUIDGen.decompose(uuid));
    }

    public static UUID readUuid(ChannelBuffer cb)
    {
        byte[] bytes = new byte[16];
        cb.readBytes(bytes);
        return UUIDGen.getUUID(ByteBuffer.wrap(bytes));
    }

    public static ChannelBuffer longStringToCB(String str)
    {
        ChannelBuffer bytes = bytes(str);
        return ChannelBuffers.wrappedBuffer(intToCB(bytes.readableBytes()), bytes);
    }

    public static List<String> readStringList(ChannelBuffer cb)
    {
        int length = cb.readUnsignedShort();
        List<String> l = new ArrayList<String>(length);
        for (int i = 0; i < length; i++)
            l.add(readString(cb));
        return l;
    }

    public static void writeStringList(ChannelBuffer cb, List<String> l)
    {
        cb.writeShort(l.size());
        for (String str : l)
            cb.writeBytes(stringToCB(str));
    }

    public static Map<String, String> readStringMap(ChannelBuffer cb)
    {
        int length = cb.readUnsignedShort();
        Map<String, String> m = new HashMap<String, String>(length);
        for (int i = 0; i < length; i++)
        {
            String k = readString(cb).toUpperCase();
            String v = readString(cb);
            m.put(k, v);
        }
        return m;
    }

    public static void writeStringMap(ChannelBuffer cb, Map<String, String> m)
    {
        cb.writeShort(m.size());
        for (Map.Entry<String, String> entry : m.entrySet())
        {
            cb.writeBytes(stringToCB(entry.getKey()));
            cb.writeBytes(stringToCB(entry.getValue()));
        }
    }

    public static Map<String, List<String>> readStringToStringListMap(ChannelBuffer cb)
    {
        int length = cb.readUnsignedShort();
        Map<String, List<String>> m = new HashMap<String, List<String>>(length);
        for (int i = 0; i < length; i++)
        {
            String k = readString(cb).toUpperCase();
            List<String> v = readStringList(cb);
            m.put(k, v);
        }
        return m;
    }

    public static void writeStringToStringListMap(ChannelBuffer cb, Map<String, List<String>> m)
    {
        cb.writeShort(m.size());
        for (Map.Entry<String, List<String>> entry : m.entrySet())
        {
            cb.writeBytes(stringToCB(entry.getKey()));
            writeStringList(cb, entry.getValue());
        }
    }

    public static ByteBuffer readValue(ChannelBuffer cb)
    {
        int length = cb.readInt();
        return length < 0 ? null : cb.readSlice(length).toByteBuffer();
    }

    public static InetSocketAddress readInet(ChannelBuffer cb)
    {
        int addrSize = cb.readByte();
        byte[] address = new byte[addrSize];
        cb.readBytes(address);
        int port = cb.readInt();
        try
        {
            return new InetSocketAddress(InetAddress.getByAddress(address), port);
        }
        catch (UnknownHostException e)
        {
            throw new ProtocolException(String.format("Invalid IP address (%d.%d.%d.%d) while deserializing inet address", address[0], address[1], address[2], address[3]));
        }
    }

    public static ChannelBuffer inetToCB(InetSocketAddress inet)
    {
        byte[] address = inet.getAddress().getAddress();
        ChannelBuffer cb = ChannelBuffers.buffer(1 + address.length + 4);
        cb.writeByte(address.length);
        cb.writeBytes(address);
        cb.writeInt(inet.getPort());
        return cb;
    }

    public static class BufferBuilder
    {
        private final int size;
        private final ChannelBuffer[] buffers;
        private int i;

        public BufferBuilder(int simpleBuffers, int stringBuffers, int valueBuffers)
        {
            this.size = simpleBuffers + 2 * stringBuffers + 2 * valueBuffers;
            this.buffers = new ChannelBuffer[size];
        }

        public BufferBuilder add(ChannelBuffer cb)
        {
            buffers[i++] = cb;
            return this;
        }

        public BufferBuilder addString(String str)
        {
            ChannelBuffer bytes = bytes(str);
            add(shortToCB(bytes.readableBytes()));
            return add(bytes);
        }

        public BufferBuilder addValue(ByteBuffer bb)
        {
            add(intToCB(bb == null ? -1 : bb.remaining()));
            return add(bb == null ? ChannelBuffers.EMPTY_BUFFER : ChannelBuffers.wrappedBuffer(bb));
        }

        public ChannelBuffer build()
        {
            return ChannelBuffers.wrappedBuffer(buffers);
        }
    }
}

redpajama_set_name: RedPajamaGithub


text:
import itertools
import json
from xlrd import open_workbook
import networkx as nx
from wikipediabase.infobox import get_meta_infobox

# READING

# add caching decorator
def read_json(path):
    """Given path to a json file, return json as python dict"""
    with open(path, 'rb') as fp:
        return json.load(fp)

def read_graph(path):
    """Given path to gpickle, return a networkx graph object"""
    return nx.read_gpickle(path)

# add caching decorator
def get_infobox_totals(path):
    """Returns a dictionary where keys are infobox templates (str) and
    values (float) are how many pages use the template. Infobox name
    of form 'Template:Infobox foo bar'.
    """
    wb = open_workbook(path)
    sheet = wb.sheet_by_index(0)

    # get the names
    names = sheet.col_values(0, 2)  # 0th col, starting at 2nd row

    # if an integer float turn into integer; else leave it alone
    int_integer = lambda name: int(name) if isinstance(
        name, float) and name.is_integer() else name

    other_names = [str(int_integer(cell)) for cell in names]

    # get the number of pages for each name
    numbers = sheet.col_values(1, 2)  # 1th col, starting at 2nd row

    # return a dictionary of form {name: number of pages}
    return dict(itertools.izip(other_names, numbers))

def get_formal_name(infobox):
    return 'Template:Infobox ' + infobox.replace('-', ' ')

def get_single_mappings(infobox):
    """Given one infobox, return its attribute mappings"""
    return get_meta_infobox(get_formal_name(infobox)).attributes

def get_all_mappings(path):
    """Given path to excel file, return dictionary with attribute mappings"""
    return _get_all_mappings(get_infoboxes(path))

def _get_all_mappings(infoboxes):
    """Given a list of infoboxes, return dictionary with attribute mappings"""
    return {infobox: get_single_mappings(infobox) for infobox in infoboxes}

def get_infoboxes(path):
    """Given path to excel file, return list of infoboxes"""
    return _get_infoboxes(get_infobox_totals(path))

def _get_infoboxes(infobox_totals):
    """Given dictionary with infobox totals, return list of infoboxes"""
    return infobox_totals.keys()

def total_infoboxes(path):
    """Given path to excel file, return total number of infoboxes"""
    return _total_infoboxes(get_infobox_totals(path))

def _total_infoboxes(infobox_totals):
    """Given dictionary with infobox totals, return total number of infoboxes"""
    return len(infobox_totals)

def total_pages(path):
    """Given path to excel file, return total number of wikipedia pages"""
    return _total_pages(get_infobox_totals(path))

def _total_pages(infobox_totals):
    """Given dictionary with infobox totals, return total number of wikipedia pages"""
    return sum(infobox_totals.values())

# WRITING

def write_json(dict, path):
    """Given a dict and a path, save dict to path as json file"""
    with open(path, 'wb') as fp:
        json.dump(dict, fp)

def write_graph(graph, path):
    """Given a graph object and a path, save graph to path as gpickle"""
    nx.write_gpickle(graph, path)


text:
package com.github.gavlyukovskiy.boot.jdbc.decorator;

import com.p6spy.engine.spy.P6DataSource;
import com.vladmihalcea.flexypool.FlexyPoolDataSource;
import com.zaxxer.hikari.HikariDataSource;
import net.ttddyy.dsproxy.support.ProxyDataSource;
import org.apache.commons.dbcp2.BasicDataSource;
import org.assertj.core.api.AbstractListAssert;
import org.assertj.core.api.ObjectAssert;
import org.junit.jupiter.api.Test;
import org.springframework.beans.DirectFieldAccessor;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Scope;
import org.springframework.context.annotation.ScopedProxyMode;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

import javax.sql.DataSource;
import java.io.PrintWriter;
import java.sql.Connection;
import java.util.List;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
import java.util.logging.Logger;

import static org.assertj.core.api.Assertions.assertThat;

class DataSourceDecoratorAutoConfigurationTests {

    private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
            .withConfiguration(AutoConfigurations.of(
                    DataSourceAutoConfiguration.class,
                    DataSourceDecoratorAutoConfiguration.class,
                    PropertyPlaceholderAutoConfiguration.class
            ))
            .withPropertyValues("spring.sql.init.mode=never",
                    "spring.datasource.url:jdbc:h2:mem:testdb-" + ThreadLocalRandom.current().nextInt());

    @Test
    void testDecoratingInDefaultOrder() {
        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);

            assertThatDataSourceDecoratingChain(dataSource).containsExactly(P6DataSource.class, ProxyDataSource.class, FlexyPoolDataSource.class);
        });
    }

    @Test
    void testNoDecoratingForExcludeBeans() {
        ApplicationContextRunner contextRunner = this.contextRunner.withPropertyValues("decorator.datasource.exclude-beans:dataSource");

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);

            assertThat(dataSource).isInstanceOf(HikariDataSource.class);
        });
    }

    @Test
    void testDecoratingWhenDefaultProxyProviderNotAvailable() {
        ApplicationContextRunner contextRunner = this.contextRunner.withClassLoader(new HidePackagesClassLoader("com.vladmihalcea.flexypool"));

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);

            assertThat(((DecoratedDataSource) dataSource).getRealDataSource()).isInstanceOf(HikariDataSource.class);
            assertThatDataSourceDecoratingChain(dataSource).containsExactly(P6DataSource.class, ProxyDataSource.class);
        });
    }

    @Test
    void testDecoratedHikariSpecificPropertiesIsSet() {
        ApplicationContextRunner contextRunner = this.contextRunner.withPropertyValues(
                "spring.datasource.type:" + HikariDataSource.class.getName(),
                "spring.datasource.hikari.catalog:test_catalog"
        );

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isNotNull();
            assertThat(dataSource).isInstanceOf(DecoratedDataSource.class);
            DataSource realDataSource = ((DecoratedDataSource) dataSource).getRealDataSource();
            assertThat(realDataSource).isInstanceOf(HikariDataSource.class);
            assertThat(((HikariDataSource) realDataSource).getCatalog()).isEqualTo("test_catalog");
        });
    }

    @Test
    void testCustomDataSourceIsDecorated() {
        ApplicationContextRunner contextRunner = this.contextRunner.withUserConfiguration(TestDataSourceConfiguration.class);

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isInstanceOf(DecoratedDataSource.class);
            DataSource realDataSource = ((DecoratedDataSource) dataSource).getRealDataSource();
            assertThat(realDataSource).isInstanceOf(BasicDataSource.class);
        });
    }

    @Test
    void testScopedDataSourceIsNotDecorated() {
        ApplicationContextRunner contextRunner = this.contextRunner.withUserConfiguration(TestScopedDataSourceConfiguration.class);

        contextRunner.run(context -> {
            assertThat(context).getBeanNames(DataSource.class).containsOnly("dataSource", "scopedTarget.dataSource");
            assertThat(context).getBean("dataSource").isInstanceOf(DecoratedDataSource.class);
            assertThat(context).getBean("scopedTarget.dataSource").isNotInstanceOf(DecoratedDataSource.class);
        });
    }

    @Test
    void testCustomDataSourceDecoratorApplied() {
        ApplicationContextRunner contextRunner = this.contextRunner.withUserConfiguration(TestDataSourceDecoratorConfiguration.class);

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isNotNull();

            DataSource customDataSource = ((DecoratedDataSource) dataSource).getDecoratedDataSource();
            assertThat(customDataSource).isInstanceOf(CustomDataSourceProxy.class);

            DataSource realDataSource = ((DecoratedDataSource) dataSource).getRealDataSource();
            assertThat(realDataSource).isInstanceOf(HikariDataSource.class);

            assertThatDataSourceDecoratingChain(dataSource).containsExactly(CustomDataSourceProxy.class, P6DataSource.class, ProxyDataSource.class,
                    FlexyPoolDataSource.class);
        });
    }

    @Test
    void testDecoratingCanBeDisabled() {
        ApplicationContextRunner contextRunner = this.contextRunner.withPropertyValues("decorator.datasource.enabled:false");

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isInstanceOf(HikariDataSource.class);
        });
    }

    @Test
    void testDecoratingCanBeDisabledForSpecificBeans() {
        ApplicationContextRunner contextRunner = this.contextRunner.withPropertyValues("decorator.datasource.exclude-beans:secondDataSource")
                .withUserConfiguration(TestMultiDataSourceConfiguration.class);

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean("dataSource", DataSource.class);
            assertThat(dataSource).isInstanceOf(DecoratedDataSource.class);

            DataSource secondDataSource = context.getBean("secondDataSource", DataSource.class);
            assertThat(secondDataSource).isInstanceOf(BasicDataSource.class);
        });
    }

    @Test
    void testDecoratingChainBuiltCorrectly() {
        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);

            DecoratedDataSource dataSource1 = context.getBean(DecoratedDataSource.class);
            assertThat(dataSource1).isNotNull();

            DataSource p6DataSource = dataSource1.getDecoratedDataSource();
            assertThat(p6DataSource).isNotNull();
            assertThat(p6DataSource).isInstanceOf(P6DataSource.class);

            DataSource proxyDataSource = (DataSource) new DirectFieldAccessor(p6DataSource)
                    .getPropertyValue("realDataSource");
            assertThat(proxyDataSource).isNotNull();
            assertThat(proxyDataSource).isInstanceOf(ProxyDataSource.class);

            DataSource flexyDataSource = (DataSource) new DirectFieldAccessor(proxyDataSource)
                    .getPropertyValue("dataSource");
            assertThat(flexyDataSource).isNotNull();
            assertThat(flexyDataSource).isInstanceOf(FlexyPoolDataSource.class);

            DataSource realDataSource = (DataSource) new DirectFieldAccessor(flexyDataSource)
                    .getPropertyValue("targetDataSource");
            assertThat(realDataSource).isNotNull();
            assertThat(realDataSource).isInstanceOf(HikariDataSource.class);

            assertThatDataSourceDecoratingChain(dataSource).containsExactly(P6DataSource.class, ProxyDataSource.class, FlexyPoolDataSource.class);

            assertThat(dataSource.toString()).isEqualTo(
                    "p6SpyDataSourceDecorator [com.p6spy.engine.spy.P6DataSource] -> " +
                            "proxyDataSourceDecorator [net.ttddyy.dsproxy.support.ProxyDataSource] -> " +
                            "flexyPoolDataSourceDecorator [com.vladmihalcea.flexypool.FlexyPoolDataSource] -> " +
                            "dataSource [com.zaxxer.hikari.HikariDataSource]");
        });
    }

    @Test
    void testDecorateDynamicallyRegisteredBeans() {
        ApplicationContextRunner contextRunner = this.contextRunner.withInitializer(context -> {
            GenericApplicationContext gac = (GenericApplicationContext) context;
            gac.registerBean("ds1", DataSource.class, () -> new HikariDataSource());
            gac.registerBean("ds2", DataSource.class, () -> new HikariDataSource());
        });

        contextRunner.run(context -> {
            DataSource dataSource1 = context.getBean("ds1", DataSource.class);
            assertThat(dataSource1).isNotNull();
            assertThat(dataSource1).isInstanceOf(DecoratedDataSource.class);

            DataSource dataSource2 = context.getBean("ds2", DataSource.class);
            assertThat(dataSource2).isNotNull();
            assertThat(dataSource2).isInstanceOf(DecoratedDataSource.class);
        });
    }

    @Test
    void testRoutingDataSourceIsDecorated() {
        ApplicationContextRunner contextRunner = this.contextRunner.withUserConfiguration(TestAbstractRoutingDataSourceConfiguration.class);

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isInstanceOf(DecoratedDataSource.class);
            DataSource realDataSource = ((DecoratedDataSource) dataSource).getRealDataSource();
            assertThat(realDataSource).isInstanceOf(AbstractRoutingDataSource.class);
        });
    }

    @Test
    void testRoutingDataSourceIsNotDecorated() {
        ApplicationContextRunner contextRunner = this.contextRunner.withPropertyValues("decorator.datasource.ignore-routing-data-sources=true")
                .withUserConfiguration(TestAbstractRoutingDataSourceConfiguration.class);

        contextRunner.run(context -> {
            DataSource dataSource = context.getBean(DataSource.class);
            assertThat(dataSource).isNotInstanceOf(DecoratedDataSource.class);
            assertThat(dataSource).isInstanceOf(AbstractRoutingDataSource.class);
        });
    }

    private AbstractListAssert<?, List<?>, Object, ObjectAssert<Object>> assertThatDataSourceDecoratingChain(DataSource dataSource) {
        return assertThat(((DecoratedDataSource) dataSource).getDecoratingChain()).extracting("dataSource").extracting("class");
    }

    @Configuration(proxyBeanMethods = false)
    static class TestDataSourceConfiguration {

        @Bean
        public DataSource dataSource() {
            BasicDataSource pool = new BasicDataSource();
            pool.setDriverClassName("org.hsqldb.jdbcDriver");
            pool.setUrl("jdbc:hsqldb:target/overridedb");
            pool.setUsername("sa");
            return pool;
        }
    }

    @Configuration(proxyBeanMethods = false)
    static class TestDataSourceDecoratorConfiguration {

        @Bean
        public DataSourceDecorator customDataSourceDecorator() {
            return (beanName, dataSource) -> new CustomDataSourceProxy(dataSource);
        }
    }

    @Configuration(proxyBeanMethods = false)
    static class TestMultiDataSourceConfiguration {

        @Bean
        @Primary
        public DataSource dataSource() {
            BasicDataSource pool = new BasicDataSource();
            pool.setDriverClassName("org.hsqldb.jdbcDriver");
            pool.setUrl("jdbc:hsqldb:target/db");
            pool.setUsername("sa");
            return pool;
        }

        @Bean
        public DataSource secondDataSource() {
            BasicDataSource pool = new BasicDataSource();
            pool.setDriverClassName("org.hsqldb.jdbcDriver");
            pool.setUrl("jdbc:hsqldb:target/db2");
            pool.setUsername("sa");
            return pool;
        }
    }

    @Configuration(proxyBeanMethods = false)
    static class TestScopedDataSourceConfiguration {

        @Bean
        @Scope(value = "prototype", proxyMode = ScopedProxyMode.TARGET_CLASS)
        public DataSource dataSource() {
            BasicDataSource pool = new BasicDataSource();
            pool.setDriverClassName("org.hsqldb.jdbcDriver");
            pool.setUrl("jdbc:hsqldb:target/overridedb");
            pool.setUsername("sa");
            return pool;
        }
    }

    @Configuration(proxyBeanMethods = false)
    static class TestAbstractRoutingDataSourceConfiguration {

        @Bean
        public DataSource dataSource() {
            AbstractRoutingDataSource routingDs = new AbstractRoutingDataSource() {
              @Override
              protected Object determineCurrentLookupKey() {
                return "ds1";
              }
            };
            BasicDataSource pool = new BasicDataSource();
            pool.setDriverClassName("org.hsqldb.jdbcDriver");
            pool.setUrl("jdbc:hsqldb:target/routingds");
            pool.setUsername("sa");
            Map<Object, Object> targetDataSources = new HashMap<>();
            targetDataSources.put("ds1", pool);
            routingDs.setTargetDataSources(targetDataSources);
            routingDs.setDefaultTargetDataSource(pool);
            return routingDs;
        }
    }

    /**
     * Custom proxy data source for tests.
     *
     * @author Arthur Gavlyukovskiy
     */
    static class CustomDataSourceProxy implements DataSource {

        private DataSource delegate;

        CustomDataSourceProxy(DataSource delegate) {
            this.delegate = delegate;
        }

        @Override
        public Connection getConnection() {
            return null;
        }

        @Override
        public Connection getConnection(String username, String password) {
            return null;
        }

        @Override
        public <T> T unwrap(Class<T> iface) {
            return null;
        }

        @Override
        public boolean isWrapperFor(Class<?> iface) {
            return false;
        }

        @Override
        public PrintWriter getLogWriter() {
            return null;
        }

        @Override
        public void setLogWriter(PrintWriter out) {

        }

        @Override
        public void setLoginTimeout(int seconds) {

        }

        @Override
        public int getLoginTimeout() {
            return 0;
        }

        @Override
        public Logger getParentLogger() {
            return null;
        }
    }
}


text:
module.exports = {

  friendlyName: 'Ensure JSON file',

  description: 'Read or write a JSON file.',

  extendedDescription: 'Assumes file is encoded using utf8.',

  cacheable: true,

  inputs: {

    path: {
      description: 'Absolute path for the JSON file (if relative path is provided, will resolve path from current working directory)',
      example: '/Users/mikermcneil/.tmp/foo.json',
      required: true
    },

    schema: {
      description: 'An example of what the parsed data should look like (used for type-coercion)',
      extendedDescription: 'If file does not exist, this schema will be used as the contents of the newly created JSON file.',
      typeclass: '*',
      required: true
    }

  },

  exits: {

    couldNotParse: {
      description: 'Could not parse file as JSON.'
    },

    success: {
      description: 'Returns the data stored in file at `source` path',
      getExample: function (inputs){
        return inputs.schema;
      }
    }

  },

  fn: function (inputs, exits) {
    var Machine = require('machine');
    var readJson = Machine.build(require('./read-json'));
    var writeJson = Machine.build(require('./write-json'));

    readJson({
      source: inputs.path,
      schema: inputs.schema
    }).exec({
      error: function (err){
        return exits.error(err);
      },
      couldNotParse: function (parseErr){
        return exits.couldNotParse(parseErr);
      },
      success: function (data){
        return exits.success(data);
      },
      doesNotExist: function (){
        // If the JSON file does not exist, create it
        writeJson({
          destination: inputs.path,
          json: inputs.schema
        }).exec({
          error: function (err){
            return exits.error(err);
          },
          success: function (){
            return exits.success(inputs.schema);
          }
        });// </writeJson>
      }//</readJson.doesNotExist>
    });//</readJson>

  }

};


text:
import theano
import theano.tensor as T
from io_func.data_io import read_data_args, read_dataset
from learn_rates import LearningRateExpDecay
from utils import parse_lrate, parse_activation, parse_conv_spec, activation_to_txt, string_2_bool

class SdAConfig():

    def __init__(self):

        # parameters related with training
        self.epochs = 5                  # number of training epochs for each layer
        self.batch_size = 128            # size of mini-batches
        self.corruption_levels=[0.2 for n in xrange(100)]  # denoising factor; we use an array for future extension to layer-specific factor
        self.learning_rates = [.01 for n in xrange(100)]   # learning rate for each layer
        self.momentum = 0                # momentum

        self.ptr_layer_number = 0        # number of layers to be trained
        self.hidden_activation = T.nnet.sigmoid   # activation function of the hidden layer/output
        self.firstlayer_reconstruct_activation = T.nnet.sigmoid      # the reconstruction activation function for the first layer
                                                                   # if we normaze the input data with mean (maybe also with variance)
                                                                   # normalization, then we need the tanh activation function to reconstruct
                                                                   # the input

        # for maxout autoencoder
        self.do_maxout = False           # whether to apply maxout on the hidden layer
        self.pool_size = 1               # pooling size of maxout

        # interfaces for the training data
        self.train_sets = None
        self.train_xy = None
        self.train_x = None
        self.train_y = None

        # interfaces for validation data. we don't do validation for RBM, so these variables will be None
        # we have these variables because we want to use the _cfg2file function from io_func/model_io.py
        self.valid_sets = None
        self.valid_xy = None
        self.valid_x = None
        self.valid_y = None

        # network structures. same as DNN
        self.n_ins = 0                  # dimension of inputs
        self.hidden_layers_sizes = []   # number of units in the hidden layers
        self.n_outs = 0                 # number of outputs (classes), not used for RBM

        # parse sparsity settings for sparsity autoencoder
        self.sparsity = None
        self.sparsity_weight = None

        # path to save model
        self.cfg_output_file = ''       # where to save this config class
        self.param_output_file = ''     # where to save the network parameters
        self.kaldi_output_file = ''     # where to save the Kaldi-formatted model

    # initialize pfile reading. TODO: inteference *directly* for Kaldi feature and alignment files
    def init_data_reading(self, train_data_spec):
        train_dataset, train_dataset_args = read_data_args(train_data_spec)
        self.train_sets, self.train_xy, self.train_x, self.train_y = read_dataset(train_dataset, train_dataset_args)

    # initialize the activation function
    def init_activation(self):
        self.activation = parse_activation(self.activation_text)

    # parse the arguments to get the values for various variables
    def parse_config_common(self, arguments):
        if arguments.has_key('corruption_level'):
            self.corruption_levels = [float(arguments['corruption_level']) for n in xrange(100)]
        if arguments.has_key('learning_rate'):
            self.learning_rates = [float(arguments['learning_rate']) for n in xrange(100)]
        if arguments.has_key('batch_size'):
            self.batch_size = int(arguments['batch_size'])
        if arguments.has_key('epoch_number'):
            self.epochs = int(arguments['epoch_number'])
        if arguments.has_key('momentum'):
            self.momentum = float(arguments['momentum'])

        # parse DNN network structure
        nnet_layers = arguments['nnet_spec'].split(':')
        self.n_ins = int(nnet_layers[0])
        self.hidden_layers_sizes = [int(nnet_layers[i]) for i in range(1, len(nnet_layers)-1)]
        self.n_outs = int(nnet_layers[-1])

        # parse pre-training layer number
        self.ptr_layer_number = len(self.hidden_layers_sizes)
        if arguments.has_key('ptr_layer_number'):
            self.ptr_layer_number = int(arguments['ptr_layer_number'])

        # parse activation function
        if arguments.has_key('hidden_activation'):
            self.hidden_activation = parse_activation(arguments['hidden_activation'])
            if arguments['hidden_activation'].startswith('maxout'):
                self.do_maxout = True; self.pool_size = int(arguments['hidden_activation'].replace('maxout:',''))
        if arguments.has_key('1stlayer_reconstruct_activation'):
            self.firstlayer_reconstruct_activation = parse_activation(arguments['1stlayer_reconstruct_activation'])

        # parse sparsity setting
        if arguments.has_key('sparsity'):
            self.sparsity = float(arguments['sparsity'])
        if arguments.has_key('sparsity_weight'):
            self.sparsity_weight = float(arguments['sparsity_weight'])

        # parse various paths for model saving
        if arguments.has_key('cfg_output_file'):
            self.cfg_output_file = arguments['cfg_output_file']
        if arguments.has_key('param_output_file'):
            self.param_output_file = arguments['param_output_file']
        if arguments.has_key('kaldi_output_file'):
            self.kaldi_output_file = arguments['kaldi_output_file']


text:
#ifndef OSMIUM_THREAD_NAME_HPP
#define OSMIUM_THREAD_NAME_HPP

#ifdef __linux__
# include <sys/prctl.h>
#endif

namespace osmium {

    namespace thread {

        /**
         * Set name of current thread for debugging. This only works on Linux.
         */
#ifdef __linux__
        inline void set_thread_name(const char* name) {
            prctl(PR_SET_NAME, name, 0, 0, 0);
        }
#else
        inline void set_thread_name(const char*) {
            // intentionally left blank
        }
#endif

    } // namespace thread

} // namespace osmium

#endif //  OSMIUM_THREAD_NAME_HPP


text:
class CloudPrintProxyBackend::Core
    : public base::RefCountedThreadSafe<CloudPrintProxyBackend::Core>,
      public CloudPrintURLFetcherDelegate,
      public cloud_print::PrintServerWatcherDelegate,
      public PrinterJobHandlerDelegate,
      public notifier::TalkMediator::Delegate {
 public:
  // It is OK for print_server_url to be empty. In this case system should
  // use system default (local) print server.
  explicit Core(CloudPrintProxyBackend* backend,
                const GURL& cloud_print_server_url,
                const DictionaryValue* print_system_settings,
                bool enable_job_poll);

  // Note:
  //
  // The Do* methods are the various entry points from CloudPrintProxyBackend
  // It calls us on a dedicated thread to actually perform synchronous
  // (and potentially blocking) operations.
  //
  // Called on the CloudPrintProxyBackend core_thread_ to perform
  // initialization. When we are passed in an LSID we authenticate using that
  // and retrieve new auth tokens.
  void DoInitializeWithLsid(const std::string& lsid,
                            const std::string& proxy_id);
  void DoInitializeWithToken(const std::string cloud_print_token,
                             const std::string cloud_print_xmpp_token,
                             const std::string email,
                             const std::string& proxy_id);

  // Called on the CloudPrintProxyBackend core_thread_ to perform
  // shutdown.
  void DoShutdown();
  void DoRegisterSelectedPrinters(
      const printing::PrinterList& printer_list);

  // CloudPrintURLFetcher::Delegate implementation.
  virtual CloudPrintURLFetcher::ResponseAction HandleJSONData(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);

  virtual void OnRequestAuthError();

  // cloud_print::PrintServerWatcherDelegate implementation
  virtual void OnPrinterAdded();
  // PrinterJobHandler::Delegate implementation
  virtual void OnPrinterJobHandlerShutdown(PrinterJobHandler* job_handler,
                                   const std::string& printer_id);
  virtual void OnAuthError();
  virtual void OnPrinterNotFound(const std::string& printer_name,
                                 bool* delete_from_server);

  // notifier::TalkMediator::Delegate implementation.
  virtual void OnNotificationStateChange(
      bool notifications_enabled);
  virtual void OnIncomingNotification(
      const notifier::Notification& notification);
  virtual void OnOutgoingNotification();

 private:
  // Prototype for a response handler.
  typedef CloudPrintURLFetcher::ResponseAction
      (CloudPrintProxyBackend::Core::*ResponseHandler)(
          const URLFetcher* source,
          const GURL& url,
          DictionaryValue* json_data,
          bool succeeded);
  // Begin response handlers
  CloudPrintURLFetcher::ResponseAction HandlePrinterListResponse(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);

  CloudPrintURLFetcher::ResponseAction HandleRegisterPrinterResponse(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);

  CloudPrintURLFetcher::ResponseAction HandleRegisterFailedStatusResponse(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);

  CloudPrintURLFetcher::ResponseAction HandlePrintSystemUnavailableResponse(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);

  CloudPrintURLFetcher::ResponseAction HandleEnumPrintersFailedResponse(
      const URLFetcher* source,
      const GURL& url,
      DictionaryValue* json_data,
      bool succeeded);
  // End response handlers

  // NotifyXXX is how the Core communicates with the frontend across
  // threads.
  void NotifyPrinterListAvailable(
      const printing::PrinterList& printer_list);
  void NotifyAuthenticated(
    const std::string& cloud_print_token,
    const std::string& cloud_print_xmpp_token,
    const std::string& email);
  void NotifyAuthenticationFailed();
  void NotifyPrintSystemUnavailable();

  // Starts a new printer registration process.
  void StartRegistration();
  // Ends the printer registration process.
  void EndRegistration();
  // Registers printer capabilities and defaults for the next printer in the
  // list with the cloud print server.
  void RegisterNextPrinter();
  // Retrieves the list of registered printers for this user/proxy combination
  // from the cloud print server.
  void GetRegisteredPrinters();
  // Removes the given printer from the list. Returns false if the printer
  // did not exist in the list.
  bool RemovePrinterFromList(const std::string& printer_name);
  // Initializes a job handler object for the specified printer. The job
  // handler is responsible for checking for pending print jobs for this
  // printer and print them.
  void InitJobHandlerForPrinter(DictionaryValue* printer_data);
  // Reports a diagnostic message to the server.
  void ReportUserMessage(const std::string& message_id,
                         const std::string& failure_message,
                         ResponseHandler handler);

  // Callback method for GetPrinterCapsAndDefaults.
  void OnReceivePrinterCaps(
      bool succeeded,
      const std::string& printer_name,
      const printing::PrinterCapsAndDefaults& caps_and_defaults);

  void HandlePrinterNotification(const std::string& printer_id);
  void PollForJobs();
  // Schedules a task to poll for jobs. Does nothing if a task is already
  // scheduled.
  void ScheduleJobPoll();

  // Our parent CloudPrintProxyBackend
  CloudPrintProxyBackend* backend_;

  GURL cloud_print_server_url_;
  scoped_ptr<DictionaryValue> print_system_settings_;
  // Pointer to current print system.
  scoped_refptr<cloud_print::PrintSystem> print_system_;
  // The list of printers to be registered with the cloud print server.
  // To begin with,this list is initialized with the list of local and network
  // printers available. Then we query the server for the list of printers
  // already registered. We trim this list to remove the printers already
  // registered. We then pass a copy of this list to the frontend to give the
  // user a chance to further trim the list. When the frontend gives us the
  // final list we make a copy into this so that we can start registering.
  printing::PrinterList printer_list_;
  // Indicates whether the printers in printer_list_ is the complete set of
  // printers to be registered for this proxy.
  bool complete_list_available_;
  // The CloudPrintURLFetcher instance for the current request.
  scoped_refptr<CloudPrintURLFetcher> request_;
  // The index of the nex printer to be uploaded.
  size_t next_upload_index_;
  // The unique id for this proxy
  std::string proxy_id_;
  // The GAIA auth token
  std::string auth_token_;
  // Cached info about the last printer that we tried to upload. We cache this
  // so we won't have to requery the printer if the upload fails and we need
  // to retry.
  std::string last_uploaded_printer_name_;
  printing::PrinterCapsAndDefaults last_uploaded_printer_info_;
  // A map of printer id to job handler.
  typedef std::map<std::string, scoped_refptr<PrinterJobHandler> >
      JobHandlerMap;
  JobHandlerMap job_handler_map_;
  ResponseHandler next_response_handler_;
  scoped_refptr<cloud_print::PrintSystem::PrintServerWatcher>
      print_server_watcher_;
  bool new_printers_available_;
  bool registration_in_progress_;
  // Notification (xmpp) handler.
  scoped_ptr<notifier::TalkMediator> talk_mediator_;
  // Indicates whether XMPP notifications are currently enabled.
  bool notifications_enabled_;
  // The time when notifications were enabled. Valid only when
  // notifications_enabled_ is true.
  base::TimeTicks notifications_enabled_since_;
  // Indicates whether a task to poll for jobs has been scheduled.
  bool job_poll_scheduled_;
  // Indicates whether we should poll for jobs when we lose XMPP connection.
  bool enable_job_poll_;
  // The channel we are interested in receiving push notifications for.
  // This is "cloudprint.google.com/proxy/<proxy_id>"
  std::string push_notifications_channel_;

  DISALLOW_COPY_AND_ASSIGN(Core);
};

CloudPrintProxyBackend::CloudPrintProxyBackend(
    CloudPrintProxyFrontend* frontend,
    const GURL& cloud_print_server_url,
    const DictionaryValue* print_system_settings,
    bool enable_job_poll)
      : core_thread_("Chrome_CloudPrintProxyCoreThread"),
        frontend_loop_(MessageLoop::current()),
        frontend_(frontend) {
  DCHECK(frontend_);
  core_ = new Core(this,
                   cloud_print_server_url,
                   print_system_settings,
                   enable_job_poll);
}

CloudPrintProxyBackend::~CloudPrintProxyBackend() {
  DCHECK(!core_);
}

bool CloudPrintProxyBackend::InitializeWithLsid(const std::string& lsid,
                                                const std::string& proxy_id) {
  if (!core_thread_.Start())
    return false;
  core_thread_.message_loop()->PostTask(FROM_HERE,
      NewRunnableMethod(
        core_.get(), &CloudPrintProxyBackend::Core::DoInitializeWithLsid, lsid,
        proxy_id));
  return true;
}

bool CloudPrintProxyBackend::InitializeWithToken(
    const std::string& cloud_print_token,
    const std::string& cloud_print_xmpp_token,
    const std::string& email,
    const std::string& proxy_id) {
  if (!core_thread_.Start())
    return false;
  core_thread_.message_loop()->PostTask(FROM_HERE,
      NewRunnableMethod(
        core_.get(), &CloudPrintProxyBackend::Core::DoInitializeWithToken,
        cloud_print_token, cloud_print_xmpp_token, email, proxy_id));
  return true;
}

void CloudPrintProxyBackend::Shutdown() {
  core_thread_.message_loop()->PostTask(FROM_HERE,
      NewRunnableMethod(core_.get(),
                        &CloudPrintProxyBackend::Core::DoShutdown));
  core_thread_.Stop();
  core_ = NULL;  // Releases reference to core_.
}

void CloudPrintProxyBackend::RegisterPrinters(
    const printing::PrinterList& printer_list) {
  core_thread_.message_loop()->PostTask(FROM_HERE,
      NewRunnableMethod(
          core_.get(),
          &CloudPrintProxyBackend::Core::DoRegisterSelectedPrinters,
          printer_list));
}

CloudPrintProxyBackend::Core::Core(CloudPrintProxyBackend* backend,
                                   const GURL& cloud_print_server_url,
                                   const DictionaryValue* print_system_settings,
                                   bool enable_job_poll)
    : backend_(backend),
      cloud_print_server_url_(cloud_print_server_url),
      complete_list_available_(false),
      next_upload_index_(0),
      next_response_handler_(NULL),
      new_printers_available_(false),
      registration_in_progress_(false),
      notifications_enabled_(false),
      job_poll_scheduled_(false),
      enable_job_poll_(enable_job_poll) {
  if (print_system_settings) {
    // It is possible to have no print settings specified.
    print_system_settings_.reset(print_system_settings->DeepCopy());
  }
}

void CloudPrintProxyBackend::Core::DoInitializeWithLsid(
    const std::string& lsid, const std::string& proxy_id) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  // Since Talk does not accept a Cloud Print token, for now, we make 2 auth
  // requests, one for the chromiumsync service and another for print. This is
  // temporary and should be removed once Talk supports our token.
  // Note: The GAIA login is synchronous but that should be OK because we are in
  // the CloudPrintProxyCoreThread and we cannot really do anything else until
  // the GAIA signin is successful.
  std::string user_agent = "ChromiumBrowser";
  scoped_refptr<ServiceGaiaAuthenticator> gaia_auth_for_talk(
      new ServiceGaiaAuthenticator(
          user_agent, kSyncGaiaServiceId, kGaiaUrl,
          g_service_process->io_thread()->message_loop_proxy()));
  gaia_auth_for_talk->set_message_loop(MessageLoop::current());
  bool auth_succeeded = false;
  if (gaia_auth_for_talk->AuthenticateWithLsid(lsid)) {
    scoped_refptr<ServiceGaiaAuthenticator> gaia_auth_for_print(
        new ServiceGaiaAuthenticator(
            user_agent, kCloudPrintGaiaServiceId, kGaiaUrl,
            g_service_process->io_thread()->message_loop_proxy()));
    gaia_auth_for_print->set_message_loop(MessageLoop::current());
    if (gaia_auth_for_print->AuthenticateWithLsid(lsid)) {
      auth_succeeded = true;
      // Let the frontend know that we have authenticated.
      backend_->frontend_loop_->PostTask(FROM_HERE, NewRunnableMethod(this,
          &Core::NotifyAuthenticated, gaia_auth_for_print->auth_token(),
          gaia_auth_for_talk->auth_token(), gaia_auth_for_talk->email()));
      backend_->frontend_loop_->PostTask(FROM_HERE, NewRunnableMethod(this,
          &Core::NotifyAuthenticated, gaia_auth_for_print->auth_token(),
          gaia_auth_for_talk->auth_token(), gaia_auth_for_talk->email()));
      DoInitializeWithToken(gaia_auth_for_print->auth_token(),
                            gaia_auth_for_talk->auth_token(),
                            gaia_auth_for_talk->email(),
                            proxy_id);
    }
  }

  if (!auth_succeeded) {
    // Let the frontend know the of authentication failure.
    backend_->frontend_loop_->PostTask(FROM_HERE, NewRunnableMethod(this,
        &Core::NotifyAuthenticationFailed));
  }
}

void CloudPrintProxyBackend::Core::DoInitializeWithToken(
    const std::string cloud_print_token,
    const std::string cloud_print_xmpp_token,
    const std::string email, const std::string& proxy_id) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Starting proxy, id: " << proxy_id;

  print_system_ =
      cloud_print::PrintSystem::CreateInstance(print_system_settings_.get());
  if (!print_system_.get()) {
    NOTREACHED();
    return;  // No print system available, fail initalization.
  }

  cloud_print::PrintSystem::PrintSystemResult result = print_system_->Init();

  // TODO(sanjeevr): Validate the tokens.
  auth_token_ = cloud_print_token;

  if (result.succeeded()) {
    notifier::NotifierOptions notifier_options;
    notifier_options.request_context_getter =
        g_service_process->GetServiceURLRequestContextGetter();
    talk_mediator_.reset(new notifier::TalkMediatorImpl(
        new notifier::MediatorThreadImpl(notifier_options),
        notifier_options));
    notifier::Subscription subscription;
    subscription.channel = kCloudPrintPushNotificationsSource;
    subscription.channel.append("/proxy/");
    subscription.channel.append(proxy_id);
    subscription.from = kCloudPrintPushNotificationsSource;
    push_notifications_channel_ = subscription.channel;
    talk_mediator_->AddSubscription(subscription);
    talk_mediator_->SetDelegate(this);
    talk_mediator_->SetAuthToken(email, cloud_print_xmpp_token,
                                 kSyncGaiaServiceId);
    talk_mediator_->Login();

    print_server_watcher_ = print_system_->CreatePrintServerWatcher();
    print_server_watcher_->StartWatching(this);

    proxy_id_ = proxy_id;

    StartRegistration();
  } else {
    // We could not initialize the print system. We need to notify the server.
    ReportUserMessage(
        kPrintSystemFailedMessageId,
        result.message(),
        &CloudPrintProxyBackend::Core::HandlePrintSystemUnavailableResponse);
  }
}

void CloudPrintProxyBackend::Core::StartRegistration() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  printer_list_.clear();
  cloud_print::PrintSystem::PrintSystemResult result =
      print_system_->EnumeratePrinters(&printer_list_);
  complete_list_available_ = result.succeeded();
  registration_in_progress_ = true;
  if (!result.succeeded()) {
    std::string message = result.message();
    if (message.empty())
      message = l10n_util::GetStringUTF8(IDS_CLOUD_PRINT_ENUM_FAILED);
    // There was a failure enumerating printers. Send a message to the server.
    ReportUserMessage(
        kEnumPrintersFailedMessageId,
        message,
        &CloudPrintProxyBackend::Core::HandleEnumPrintersFailedResponse);
  } else {
    // Now we need to ask the server about printers that were registered on the
    // server so that we can trim this list.
    GetRegisteredPrinters();
  }
}

void CloudPrintProxyBackend::Core::EndRegistration() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  request_ = NULL;
  registration_in_progress_ = false;
  if (new_printers_available_) {
    new_printers_available_ = false;
    StartRegistration();
  }
}

void CloudPrintProxyBackend::Core::DoShutdown() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Shutdown proxy, id: " << proxy_id_;
  if (print_server_watcher_ != NULL)
    print_server_watcher_->StopWatching();

  // Need to kill all running jobs.
  while (!job_handler_map_.empty()) {
    JobHandlerMap::iterator index = job_handler_map_.begin();
    // Shutdown will call our OnPrinterJobHandlerShutdown method which will
    // remove this from the map.
    index->second->Shutdown();
  }
  // Important to delete the TalkMediator on this thread.
  talk_mediator_->Logout();
  talk_mediator_.reset();
  notifications_enabled_ = false;
  notifications_enabled_since_ = base::TimeTicks();
  request_ = NULL;
}

void CloudPrintProxyBackend::Core::DoRegisterSelectedPrinters(
    const printing::PrinterList& printer_list) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  if (!print_system_.get())
    return;  // No print system available.
  printer_list_.assign(printer_list.begin(), printer_list.end());
  next_upload_index_ = 0;
  RegisterNextPrinter();
}

void CloudPrintProxyBackend::Core::GetRegisteredPrinters() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  GURL printer_list_url =
      CloudPrintHelpers::GetUrlForPrinterList(cloud_print_server_url_,
                                              proxy_id_);
  next_response_handler_ =
      &CloudPrintProxyBackend::Core::HandlePrinterListResponse;
  request_ = new CloudPrintURLFetcher;
  request_->StartGetRequest(printer_list_url,
                            this,
                            auth_token_,
                            kCloudPrintAPIMaxRetryCount,
                            std::string());
}

void CloudPrintProxyBackend::Core::RegisterNextPrinter() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  // For the next printer to be uploaded, create a multi-part post request to
  // upload the printer capabilities and the printer defaults.
  if (next_upload_index_ < printer_list_.size()) {
    const printing::PrinterBasicInfo& info =
        printer_list_.at(next_upload_index_);
    // If we are retrying a previous upload, we don't need to fetch the caps
    // and defaults again.
    if (info.printer_name != last_uploaded_printer_name_) {
      cloud_print::PrintSystem::PrinterCapsAndDefaultsCallback* callback =
           NewCallback(this,
                       &CloudPrintProxyBackend::Core::OnReceivePrinterCaps);
        // Asnchronously fetch the printer caps and defaults. The story will
        // continue in OnReceivePrinterCaps.
        print_system_->GetPrinterCapsAndDefaults(
            info.printer_name.c_str(), callback);
    } else {
      OnReceivePrinterCaps(true,
                           last_uploaded_printer_name_,
                           last_uploaded_printer_info_);
    }
  } else {
    EndRegistration();
  }
}

void CloudPrintProxyBackend::Core::OnReceivePrinterCaps(
    bool succeeded,
    const std::string& printer_name,
    const printing::PrinterCapsAndDefaults& caps_and_defaults) {
  DCHECK(next_upload_index_ < printer_list_.size());
  if (succeeded) {
    const printing::PrinterBasicInfo& info =
        printer_list_.at(next_upload_index_);

    last_uploaded_printer_name_ = info.printer_name;
    last_uploaded_printer_info_ = caps_and_defaults;

    std::string mime_boundary;
    CloudPrintHelpers::CreateMimeBoundaryForUpload(&mime_boundary);
    std::string post_data;

    CloudPrintHelpers::AddMultipartValueForUpload(kProxyIdValue, proxy_id_,
                                                  mime_boundary,
                                                  std::string(), &post_data);
    CloudPrintHelpers::AddMultipartValueForUpload(kPrinterNameValue,
                                                  info.printer_name,
                                                  mime_boundary,
                                                  std::string(), &post_data);
    CloudPrintHelpers::AddMultipartValueForUpload(kPrinterDescValue,
                                                  info.printer_description,
                                                  mime_boundary,
                                                  std::string() , &post_data);
    CloudPrintHelpers::AddMultipartValueForUpload(
        kPrinterStatusValue, StringPrintf("%d", info.printer_status),
        mime_boundary, std::string(), &post_data);
    // Add printer options as tags.
    CloudPrintHelpers::GenerateMultipartPostDataForPrinterTags(info.options,
                                                               mime_boundary,
                                                               &post_data);

    CloudPrintHelpers::AddMultipartValueForUpload(
        kPrinterCapsValue, last_uploaded_printer_info_.printer_capabilities,
        mime_boundary, last_uploaded_printer_info_.caps_mime_type,
        &post_data);
    CloudPrintHelpers::AddMultipartValueForUpload(
        kPrinterDefaultsValue, last_uploaded_printer_info_.printer_defaults,
        mime_boundary, last_uploaded_printer_info_.defaults_mime_type,
        &post_data);
    // Send a hash of the printer capabilities to the server. We will use this
    // later to check if the capabilities have changed
    CloudPrintHelpers::AddMultipartValueForUpload(
        kPrinterCapsHashValue,
        MD5String(last_uploaded_printer_info_.printer_capabilities),
        mime_boundary, std::string(), &post_data);
    GURL post_url = CloudPrintHelpers::GetUrlForPrinterRegistration(
        cloud_print_server_url_);

    next_response_handler_ =
        &CloudPrintProxyBackend::Core::HandleRegisterPrinterResponse;
    // Terminate the request body
    post_data.append("--" + mime_boundary + "--\r\n");
    std::string mime_type("multipart/form-data; boundary=");
    mime_type += mime_boundary;
    request_ = new CloudPrintURLFetcher;
    request_->StartPostRequest(post_url,
                               this,
                               auth_token_,
                               kCloudPrintAPIMaxRetryCount,
                               mime_type,
                               post_data,
                               std::string());
  } else {
    LOG(ERROR) << "CP_PROXY: Failed to get printer info for: " <<
        printer_name;
    // This printer failed to register, notify the server of this failure.
    string16 printer_name_utf16 = UTF8ToUTF16(printer_name);
    std::string status_message = l10n_util::GetStringFUTF8(
        IDS_CLOUD_PRINT_REGISTER_PRINTER_FAILED,
        printer_name_utf16);
    ReportUserMessage(
        kGetPrinterCapsFailedMessageId,
        status_message,
        &CloudPrintProxyBackend::Core::HandleRegisterFailedStatusResponse);
  }
}

void CloudPrintProxyBackend::Core::HandlePrinterNotification(
    const std::string& printer_id) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Handle printer notification, id: " << printer_id;
  JobHandlerMap::iterator index = job_handler_map_.find(printer_id);
  if (index != job_handler_map_.end())
    index->second->CheckForJobs(kJobFetchReasonNotified);
}

void CloudPrintProxyBackend::Core::PollForJobs() {
  VLOG(1) << "CP_PROXY: Polling for jobs.";
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  for (JobHandlerMap::iterator index = job_handler_map_.begin();
       index != job_handler_map_.end(); index++) {
    // If notifications are on, then we should poll for this printer only if
    // the last time it fetched jobs was before notifications were last enabled.
    bool should_poll =
        !notifications_enabled_ ||
        (index->second->last_job_fetch_time() <= notifications_enabled_since_);
    if (should_poll)
      index->second->CheckForJobs(kJobFetchReasonPoll);
  }
  job_poll_scheduled_ = false;
  // If we don't have notifications and job polling is enabled, poll again
  // after a while.
  if (!notifications_enabled_ && enable_job_poll_)
    ScheduleJobPoll();
}

void CloudPrintProxyBackend::Core::ScheduleJobPoll() {
  if (!job_poll_scheduled_) {
    int interval_in_seconds = base::RandInt(kMinJobPollIntervalSecs,
                                            kMaxJobPollIntervalSecs);
    MessageLoop::current()->PostDelayedTask(
        FROM_HERE,
        NewRunnableMethod(this, &CloudPrintProxyBackend::Core::PollForJobs),
        interval_in_seconds * 1000);
    job_poll_scheduled_ = true;
  }
}

// CloudPrintURLFetcher::Delegate implementation.
CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandleJSONData(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(next_response_handler_);
  return (this->*next_response_handler_)(source, url, json_data, succeeded);
}

void CloudPrintProxyBackend::Core::OnRequestAuthError() {
  OnAuthError();
}

void CloudPrintProxyBackend::Core::NotifyPrinterListAvailable(
    const printing::PrinterList& printer_list) {
  DCHECK(MessageLoop::current() == backend_->frontend_loop_);
  backend_->frontend_->OnPrinterListAvailable(printer_list);
}

void CloudPrintProxyBackend::Core::NotifyAuthenticated(
    const std::string& cloud_print_token,
    const std::string& cloud_print_xmpp_token,
    const std::string& email) {
  DCHECK(MessageLoop::current() == backend_->frontend_loop_);
  backend_->frontend_->OnAuthenticated(cloud_print_token,
                                       cloud_print_xmpp_token, email);
}

void CloudPrintProxyBackend::Core::NotifyAuthenticationFailed() {
  DCHECK(MessageLoop::current() == backend_->frontend_loop_);
  backend_->frontend_->OnAuthenticationFailed();
}

void CloudPrintProxyBackend::Core::NotifyPrintSystemUnavailable() {
  DCHECK(MessageLoop::current() == backend_->frontend_loop_);
  backend_->frontend_->OnPrintSystemUnavailable();
}

CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandlePrinterListResponse(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  if (!succeeded) {
    NOTREACHED();
    return CloudPrintURLFetcher::RETRY_REQUEST;
  }
  ListValue* printer_list = NULL;
  json_data->GetList(kPrinterListValue, &printer_list);
  // There may be no "printers" value in the JSON
  if (printer_list) {
    for (size_t index = 0; index < printer_list->GetSize(); index++) {
      DictionaryValue* printer_data = NULL;
      if (printer_list->GetDictionary(index, &printer_data)) {
        std::string printer_name;
        printer_data->GetString(kNameValue, &printer_name);
        RemovePrinterFromList(printer_name);
        InitJobHandlerForPrinter(printer_data);
      } else {
        NOTREACHED();
      }
    }
  }
  request_ = NULL;
  if (!printer_list_.empty()) {
    // Let the frontend know that we have a list of printers available.
    backend_->frontend_loop_->PostTask(FROM_HERE, NewRunnableMethod(this,
        &Core::NotifyPrinterListAvailable, printer_list_));
  } else {
    // No more work to be done here.
    MessageLoop::current()->PostTask(
        FROM_HERE, NewRunnableMethod(this, &Core::EndRegistration));
  }
  return CloudPrintURLFetcher::STOP_PROCESSING;
}

void CloudPrintProxyBackend::Core::InitJobHandlerForPrinter(
    DictionaryValue* printer_data) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  DCHECK(printer_data);
  PrinterJobHandler::PrinterInfoFromCloud printer_info_cloud;
  printer_data->GetString(kIdValue, &printer_info_cloud.printer_id);
  DCHECK(!printer_info_cloud.printer_id.empty());
  VLOG(1) << "CP_PROXY: Init job handler for printer id: "
          << printer_info_cloud.printer_id;
  JobHandlerMap::iterator index = job_handler_map_.find(
      printer_info_cloud.printer_id);
  // We might already have a job handler for this printer
  if (index == job_handler_map_.end()) {
    printing::PrinterBasicInfo printer_info;
    printer_data->GetString(kNameValue, &printer_info.printer_name);
    DCHECK(!printer_info.printer_name.empty());
    printer_data->GetString(kPrinterDescValue,
                            &printer_info.printer_description);
    // Printer status is a string value which actually contains an integer.
    std::string printer_status;
    if (printer_data->GetString(kPrinterStatusValue, &printer_status)) {
      base::StringToInt(printer_status, &printer_info.printer_status);
    }
    printer_data->GetString(kPrinterCapsHashValue,
        &printer_info_cloud.caps_hash);
    ListValue* tags_list = NULL;
    printer_data->GetList(kTagsValue, &tags_list);
    if (tags_list) {
      for (size_t index = 0; index < tags_list->GetSize(); index++) {
        std::string tag;
        tags_list->GetString(index, &tag);
        if (StartsWithASCII(tag, kTagsHashTagName, false)) {
          std::vector<std::string> tag_parts;
          base::SplitStringDontTrim(tag, '=', &tag_parts);
          DCHECK_EQ(tag_parts.size(), 2U);
          if (tag_parts.size() == 2)
            printer_info_cloud.tags_hash = tag_parts[1];
        }
      }
    }
    scoped_refptr<PrinterJobHandler> job_handler;
    job_handler = new PrinterJobHandler(printer_info, printer_info_cloud,
                                        auth_token_, cloud_print_server_url_,
                                        print_system_.get(), this);
    job_handler_map_[printer_info_cloud.printer_id] = job_handler;
    job_handler->Initialize();
  }
}

void CloudPrintProxyBackend::Core::ReportUserMessage(
    const std::string& message_id,
    const std::string& failure_message,
    ResponseHandler handler) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  std::string mime_boundary;
  CloudPrintHelpers::CreateMimeBoundaryForUpload(&mime_boundary);
  GURL post_url = CloudPrintHelpers::GetUrlForUserMessage(
      cloud_print_server_url_,
      message_id);
  std::string post_data;
  CloudPrintHelpers::AddMultipartValueForUpload(kMessageTextValue,
                                                failure_message,
                                                mime_boundary,
                                                std::string(),
                                                &post_data);
  next_response_handler_ = handler;
  // Terminate the request body
  post_data.append("--" + mime_boundary + "--\r\n");
  std::string mime_type("multipart/form-data; boundary=");
  mime_type += mime_boundary;
  request_ = new CloudPrintURLFetcher;
  request_->StartPostRequest(post_url,
                             this,
                             auth_token_,
                             kCloudPrintAPIMaxRetryCount,
                             mime_type,
                             post_data,
                             std::string());
}

CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandleRegisterPrinterResponse(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  if (succeeded) {
    ListValue* printer_list = NULL;
    json_data->GetList(kPrinterListValue, &printer_list);
    // There should be a "printers" value in the JSON
    DCHECK(printer_list);
    if (printer_list) {
      DictionaryValue* printer_data = NULL;
      if (printer_list->GetDictionary(0, &printer_data))
        InitJobHandlerForPrinter(printer_data);
    }
  }
  next_upload_index_++;
  MessageLoop::current()->PostTask(
      FROM_HERE,
      NewRunnableMethod(this,
                        &CloudPrintProxyBackend::Core::RegisterNextPrinter));
  return CloudPrintURLFetcher::STOP_PROCESSING;
}

CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandleRegisterFailedStatusResponse(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  next_upload_index_++;
  MessageLoop::current()->PostTask(
      FROM_HERE,
      NewRunnableMethod(this,
                        &CloudPrintProxyBackend::Core::RegisterNextPrinter));
  return CloudPrintURLFetcher::STOP_PROCESSING;
}

CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandlePrintSystemUnavailableResponse(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  // Let the frontend know that we do not have a print system.
  backend_->frontend_loop_->PostTask(
      FROM_HERE,
      NewRunnableMethod(this,
                        &Core::NotifyPrintSystemUnavailable));
  return CloudPrintURLFetcher::STOP_PROCESSING;
}

CloudPrintURLFetcher::ResponseAction
CloudPrintProxyBackend::Core::HandleEnumPrintersFailedResponse(
    const URLFetcher* source,
    const GURL& url,
    DictionaryValue* json_data,
    bool succeeded) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  // Now proceed with printer registration.
  GetRegisteredPrinters();
  return CloudPrintURLFetcher::STOP_PROCESSING;
}

bool CloudPrintProxyBackend::Core::RemovePrinterFromList(
    const std::string& printer_name) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  for (printing::PrinterList::iterator index = printer_list_.begin();
       index != printer_list_.end(); index++) {
    if (0 == base::strcasecmp(index->printer_name.c_str(),
                              printer_name.c_str())) {
      index = printer_list_.erase(index);
      return true;
    }
  }
  return false;
}

void CloudPrintProxyBackend::Core::OnNotificationStateChange(
    bool notification_enabled) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  notifications_enabled_ = notification_enabled;
  if (notifications_enabled_) {
    notifications_enabled_since_ = base::TimeTicks::Now();
    VLOG(1) << "Notifications for proxy " << proxy_id_ << " were enabled at "
            << notifications_enabled_since_.ToInternalValue();
  } else {
    LOG(ERROR) << "Notifications for proxy " << proxy_id_ << " disabled.";
    notifications_enabled_since_ = base::TimeTicks();
  }
  // A state change means one of two cases.
  // Case 1: We just lost notifications. This this case we want to schedule a
  // job poll if enable_job_poll_ is true.
  // Case 2: Notifications just got re-enabled. In this case we want to schedule
  // a poll once for jobs we might have missed when we were dark.
  // Note that ScheduleJobPoll will not schedule again if a job poll task is
  // already scheduled.
  if (enable_job_poll_ || notifications_enabled_)
    ScheduleJobPoll();
}

void CloudPrintProxyBackend::Core::OnIncomingNotification(
    const notifier::Notification& notification) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Incoming notification.";
  if (0 == base::strcasecmp(push_notifications_channel_.c_str(),
                            notification.channel.c_str()))
    HandlePrinterNotification(notification.data);
}

void CloudPrintProxyBackend::Core::OnOutgoingNotification() {}

// cloud_print::PrinterChangeNotifier::Delegate implementation
void CloudPrintProxyBackend::Core::OnPrinterAdded() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  if (registration_in_progress_)
    new_printers_available_ = true;
  else
    StartRegistration();
}

// PrinterJobHandler::Delegate implementation
void CloudPrintProxyBackend::Core::OnPrinterJobHandlerShutdown(
    PrinterJobHandler* job_handler, const std::string& printer_id) {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Printer job handle shutdown, id " << printer_id;
  job_handler_map_.erase(printer_id);
}

void CloudPrintProxyBackend::Core::OnAuthError() {
  DCHECK(MessageLoop::current() == backend_->core_thread_.message_loop());
  VLOG(1) << "CP_PROXY: Auth Error";
  backend_->frontend_loop_->PostTask(FROM_HERE, NewRunnableMethod(this,
      &Core::NotifyAuthenticationFailed));
}

void CloudPrintProxyBackend::Core::OnPrinterNotFound(
    const std::string& printer_name,
    bool* delete_from_server) {
  // If we have a complete list of local printers, then this needs to be deleted
  // from the server.
  *delete_from_server = complete_list_available_;
}


text:
require 'angus'

module Spec
  module Functional

    class TypeValidation < Angus::Base
      def configure
        register :admins
      end
    end

  end
end


text:
module Types
    class SavedSearchType < Types::BaseObject
      field :id, Integer, null: false
      field :user_id, Integer, null: false
      field :url, String, null: false
      field :updated_at, GraphQL::Types::ISO8601DateTime, null:false
      field :created_at, GraphQL::Types::ISO8601DateTime, null:false
      field :short_link, ShortLinkType, null:false
      field :name_label, String, null:true
      field :is_subscribed, Boolean, null:false
    end
  end


text:
####Compiling TypeScript with npm and starting an HTTP server

1. Run "npm install" at the command-line from the root of the project (where the package.json file lives).
2. Run "npm start" at the command-line. This will compile all of the TypeScript files and generate .js and .map (for debugging) files.
A watcher will also be created to monitor any changes in the .ts files. Leave the command-window open while doing development work.
3. Open the project folder in VS Code (http://code.visualstudio.com) or another editor.

####Compiling TypeScript with Gulp:

If you'd prefer to compile with Gulp do the following:

1. Run "npm install" at the command-line from the root of the project (where the package.json file lives).
2. Run "gulp" at the command-line. This will compile all of the TypeScript files and generate .js and .map (for debugging) files.
3. Open the project folder in VS Code (http://code.visualstudio.com) or another editor.