text:
# connect-mongo

MongoDB session store for [Connect](https://github.com/senchalabs/connect) and [Express](http://expressjs.com/)

[![Build Status](https://travis-ci.org/kcbanner/connect-mongo.svg?branch=master)](https://travis-ci.org/kcbanner/connect-mongo) [![Dependency Status](https://david-dm.org/kcbanner/connect-mongo.svg?style=flat)](https://david-dm.org/kcbanner/connect-mongo)

## Compatibility

* Support Express `4.x`, `5.0` and Connect `3.x` through [express-session](https://github.com/expressjs/session)
* Support Express `2.x`, `3.x` and Connect `>= 1.0.3`, `2.x`
* Support [Mongoose](http://mongoosejs.com/index.html) `>= 2.6` and `3.x`
* Support [native MongoDB driver](http://mongodb.github.io/node-mongodb-native/) `>= 1.2` and `2.0`
* Support Node.js `0.8`, `0.10` and `0.11`

## Usage

### Express or Connect integration

Express `4.x`, `5.0` and Connect `3.x`:

```js
var session = require('express-session');
var MongoStore = require('connect-mongo')(session);

app.use(session({
    secret: 'foo',
    store: new MongoStore(options)
}));
```

Express `2.x`, `3.x` and Connect `1.x`, `2.x`:

```js
var MongoStore = require('connect-mongo')(express);

app.use(express.session({
    secret: 'foo',
    store: new MongoStore(options)
}));
```

For Connect `1.x` and `2.x`, just replace `express` by `connect`.

### Connection to MongoDB

In many circumstances, `connect-mongo` will not be the only part of your application which need a connection to a MongoDB database. It could be interesting to re-use an existing connection.

Alternatively, you can configure `connect-mongo` to establish a new connection.

#### Re-use a Mongoose connection

```js
var mongoose = require('mongoose');

// Basic usage
mongoose.connect(connectionOptions);

app.use(session({
    store: new MongoStore({ mongooseConnection: mongoose.connection })
}));

// Advanced usage
var connection = mongoose.createConnection(connectionOptions);

app.use(session({
    store: new MongoStore({ mongooseConnection: connection })
}));
```

#### Re-use a native MongoDB driver connection

In this case, you juste have to give your `Db` instance to `connect-mongo`.
If the connection is not opened, `connect-mongo` will do it for you.

```js
/*
** There are many ways to create dbInstance.
** You should refer to the driver documentation.
*/

app.use(session({
    store: new MongoStore({ db: dbInstance })
}));
```

#### Create a new connection from a MongoDB connection string

[MongoDB connection strings](http://docs.mongodb.org/manual/reference/connection-string/) are __the best way__ to configure a new connection. For advanced usage, [more options](http://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html#mongoclient-connect-options) can be configured with `mongoOptions` property.

```js
// Basic usage
app.use(session({
    store: new MongoStore({ url: 'mongodb://localhost/test-app' })
}));

// Advanced usage
app.use(session({
    store: new MongoStore({
        url: 'mongodb://user12345:foobar@localhost/test-app?authSource=admins&w=1',
        mongoOptions: advancedOptions // See below for details
    })
}));
```

#### Create a new connection from legacy options

For compatibility purpose, `connect-mongo` can create a basic connection string and set some options for you.

You can't use `mongoOptions` if you choose this mode.

```js
app.use(session({
    store: new MongoStore({
        // Basic usage
        host: 'localhost', // Default, optional
        port: 27017, // Default, optional
        db: 'test-app', // Required

        // Basic authentication (optional)
        username: 'user12345',
        password: 'foobar',

        // Advanced options (optional)
        autoReconnect: true, // Default
        w: 1, // Default,
        ssl: false // Default
    })
}));
```

## More options

  - `collection` Collection (optional, default: `sessions`)
  - `stringify` If true, connect-mongo will serialize sessions using `JSON.stringify` before
                setting them, and deserialize them with `JSON.parse` when getting them.
                (optional, default: true). This is useful if you are using types that
                MongoDB doesn't support.
  - `serialize` Custom hook for serializing sessions to MongoDB. This is helpful if you need
                to modify the session before writing it out.
  - `unserialize` Custom hook for unserializing sessions from MongoDB. This can be used in
                scenarios where you need to support different types of serializations
                (e.g., objects and JSON strings) or need to modify the session before using
                it in your app.
  - `hash` (optional) Hash is an object, which will determine wether hash the sid in mongo, since it's not undefined, means sid will be hashed
  - `hash.salt` Salt will be used to hash the sid in mongo, default salt is "connect-mongo"
  - `hash.algorithm` Hash algorithm, default algorithm is "sha1"

## Removing expired sessions

  connect-mongo uses MongoDB's TTL collection feature (2.2+) to
  have mongod automatically remove expired sessions. (mongod runs this
  check every minute.)

  **Note:** By connect/express's default, session cookies are set to
  expire when the user closes their browser (maxAge: null). In accordance
  with standard industry practices, connect-mongo will set these sessions
  to expire two weeks from their last 'set'. You can override this
  behavior by manually setting the maxAge for your cookies -- just keep in
  mind that any value less than 60 seconds is pointless, as mongod will
  only delete expired documents in a TTL collection every minute.

## Tests

You need `mocha`.

    make test

The tests use a database called `connect-mongo-test`.

## License

(The MIT License)

Copyright (c) 2011 Casey Banner <kcbanner@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


text:
"use strict";

const should = require("should");

const Model = require("../lib/model");
const Query = require("../lib/query");
const Toshihiko = require("../lib/toshihiko");
const Yukari = require("../lib/yukari");

describe("🐣 model", function() {
    const toshihiko = new Toshihiko("base");
    const model = new Model("name", toshihiko, [ { name: "key1" } ]);

    describe("define", function() {
        it("should define with cache", function() {
            const model = new Model("name", toshihiko, [
                { name: "key1" }
            ], {
                cache: {
                    module: require("./util/cache")
                }
            });

            model.should.match({
                ai: null,
                primaryKeys: [],
                name: "name",
                schema: [{
                    name: "key1",
                    column: "key1",
                    primaryKey: false
                }],
                cache: {
                    foo: undefined,
                    bar: undefined
                }
            });
        });

        it("should define with parent's cache", function() {
            const toshihiko = new Toshihiko("base", {
                cache: {
                    module: require("./util/cache"),
                    foo: 1,
                    bar: 2
                }
            });
            const model = new Model("name", toshihiko, [
                { name: "key1" }
            ]);

            model.should.match({
                ai: null,
                primaryKeys: [],
                name: "name",
                schema: [{
                    name: "key1",
                    column: "key1",
                    primaryKey: false
                }],
                cache: {
                    foo: 1,
                    bar: 2
                }
            });
        });

        it("should define without", function() {
            const toshihiko = new Toshihiko("base", {
                cache: {
                    module: require("./util/cache"),
                    foo: 1,
                    bar: 2
                }
            });
            const model = new Model("name", toshihiko, [
                { name: "key1" }
            ]);

            model.should.match({
                ai: null,
                primaryKeys: [],
                name: "name",
                schema: [{
                    name: "key1",
                    column: "key1",
                    primaryKey: false
                }],
                cache: {
                    foo: 1,
                    bar: 2
                }
            });

            model.toshihiko.should.equal(toshihiko);
        });

        it("should have getters", function() {
            model._fields.should.equal(model.schema);
        });
    });

    it("should build", function() {
        const yukari = new Yukari(model, "new");
        yukari.buildNewRow({ key1: 0 });
        yukari.should.deepEqual(model.build({ key1: 0 }));
    });

    describe("queries", function() {
        function test(name, argu) {
            it(name, function() {
                const query = new Query(model);
                model[name].apply(model, argu).should.deepEqual(query[name].apply(query, argu));
            });
        }

        test("where", [{ foo: "bar" }]);
        test("fields", [ "key1" ]);
        test("field", [ "key1" ]);
        test("limit", [ 1, 2 ]);
        test("limit", [ 1 ]);
        test("order", [{ key1: 1 }]);
        test("orderBy", [{ key1: -1 }]);
        test("index", [ "idx" ]);

        it("count", function(done) {
            model.count(function(err) {
                err.message.should.equal("this adapter's count function is not implemented yet.");
                done();
            });
        });

        it("find", function(done) {
            model.find(function(err) {
                err.message.should.equal("this adapter's find function is not implemented yet.");
                done();
            }, true, true);
        });

        it("findById", function(done) {
            model.findById("123", function(err) {
                err.message.should.equal("you should pass a valid IDs object");
                done();
            }, true);
        });

        it("findOne", function(done) {
            model.findOne(function(err) {
                err.message.should.equal("this adapter's find function is not implemented yet.");
                done();
            }, true);
        });

        it("update", function(done) {
            model.update({ foo: "bar" }, function(err) {
                err.message.should.equal("this adapter's updateByQuery function is not implemented yet.");
                done();
            });
        });

        it("delete", function(done) {
            model.delete(function(err) {
                err.message.should.equal("this adapter's deleteByQuery function is not implemented yet.");
                done();
            });
        });

        it("execute", function(done) {
            model.execute(function(err) {
                err.message.should.equal("this adapter's execute function is not implemented yet.");
                done();
            });
        });
    });

    describe("convertColumnToName", function() {
        const model = new Model("name", toshihiko, require("./util/common.js").COMMON_SCHEMA);

        it("pass string parameter", function() {
            model.convertColumnToName("id").should.equal("key1");
        });

        it("pass array parameter", function() {
            model.convertColumnToName([ "id", "key2" ]).should.deepEqual([ "key1", "key2" ]);
        });

        it("pass object parameter", function() {
            model.convertColumnToName({
                id: 1,
                key2: 2
            }).should.deepEqual({
                key1: 1,
                key2: 2
            });
        });

        it("should return undefined", function() {
            should(model.convertColumnToName(1)).equal(undefined);
        });
    });

    describe("compatible", function() {
        const model1 = new Model("base", toshihiko, require("./util/common").COMMON_SCHEMA);
        const model2 = new Model("base", toshihiko, require("./util/common").COMMON_SCHEMA_NO_PRIMARY);
        const model3 = new Model("base", toshihiko, require("./util/common").COMMON_SCHEMA_MULTI_PRIMARY);

        describe("getPrimaryKeysName", function() {
            it("no primary key", function() {
                model2.getPrimaryKeysName().should.deepEqual([]);
            });

            it("single primary key", function() {
                model1.getPrimaryKeysName().should.deepEqual("key1");
            });

            it("multiple primary keys", function() {
                model3.getPrimaryKeysName().should.deepEqual([ "key1", "key4" ]);
            });
        });

        describe("getPrimaryKeysColumn", function() {
            it("no primary key", function() {
                model2.getPrimaryKeysColumn().should.deepEqual([]);
            });

            it("single primary key", function() {
                model1.getPrimaryKeysColumn().should.deepEqual("id");
            });

            it("multiple primary keys", function() {
                model3.getPrimaryKeysColumn().should.deepEqual([ "id", "key4" ]);
            });
        });
    });
});

redpajama_set_name: RedPajamaGithub


text:
package kubeadm

import (
	"bytes"
	"crypto"
	"fmt"
	"io"
	"os"
	"path"
	"strings"
	"time"

	"github.com/docker/machine/libmachine"
	"github.com/docker/machine/libmachine/state"
	"github.com/golang/glog"
	download "github.com/jimmidyson/go-download"
	"github.com/pkg/errors"
	"golang.org/x/sync/errgroup"
	"k8s.io/minikube/pkg/minikube/assets"
	"k8s.io/minikube/pkg/minikube/bootstrapper"
	"k8s.io/minikube/pkg/minikube/config"
	"k8s.io/minikube/pkg/minikube/constants"
	"k8s.io/minikube/pkg/minikube/machine"
	"k8s.io/minikube/pkg/minikube/sshutil"
	"k8s.io/minikube/pkg/util"
)

type KubeadmBootstrapper struct {
	c bootstrapper.CommandRunner
}

func NewKubeadmBootstrapper(api libmachine.API) (*KubeadmBootstrapper, error) {
	h, err := api.Load(config.GetMachineName())
	if err != nil {
		return nil, errors.Wrap(err, "getting api client")
	}
	var cmd bootstrapper.CommandRunner
	// The none driver executes commands directly on the host
	if h.Driver.DriverName() == constants.DriverNone {
		cmd = &bootstrapper.ExecRunner{}
	} else {
		client, err := sshutil.NewSSHClient(h.Driver)
		if err != nil {
			return nil, errors.Wrap(err, "getting ssh client")
		}
		cmd = bootstrapper.NewSSHRunner(client)
	}
	return &KubeadmBootstrapper{
		c: cmd,
	}, nil
}

//TODO(r2d4): This should most likely check the health of the apiserver
func (k *KubeadmBootstrapper) GetClusterStatus() (string, error) {
	statusCmd := `sudo systemctl is-active kubelet &>/dev/null && echo "Running" || echo "Stopped"`
	status, err := k.c.CombinedOutput(statusCmd)
	if err != nil {
		return "", errors.Wrap(err, "getting status")
	}
	status = strings.TrimSpace(status)
	if status == state.Running.String() || status == state.Stopped.String() {
		return status, nil
	}
	return "", fmt.Errorf("Error: Unrecognized output from ClusterStatus: %s", status)
}

// TODO(r2d4): Should this aggregate all the logs from the control plane?
// Maybe subcommands for each component? minikube logs apiserver?
func (k *KubeadmBootstrapper) GetClusterLogsTo(follow bool, out io.Writer) error {
	var flags []string
	if follow {
		flags = append(flags, "-f")
	}
	logsCommand := fmt.Sprintf("sudo journalctl %s -u kubelet", strings.Join(flags, " "))

	if follow {
		if err := k.c.CombinedOutputTo(logsCommand, out); err != nil {
			return errors.Wrap(err, "getting cluster logs")
		}
	} else {

		logs, err := k.c.CombinedOutput(logsCommand)
		if err != nil {
			return errors.Wrap(err, "getting cluster logs")
		}
		fmt.Fprint(out, logs)
	}
	return nil
}

func (k *KubeadmBootstrapper) StartCluster(k8s bootstrapper.KubernetesConfig) error {
	// We use --ignore-preflight-errors=DirAvailable since we have our own custom addons
	// that we also stick in /etc/kubernetes/manifests
	// We use --ignore-preflight-errors=Swap since minikube.iso allocates a swap partition.
	// (it should probably stop doing this, though...)
	b := bytes.Buffer{}
	if err := kubeadmInitTemplate.Execute(&b, struct{ KubeadmConfigFile string }{constants.KubeadmConfigFile}); err != nil {
		return err
	}

	err := k.c.Run(b.String())
	if err != nil {
		return errors.Wrapf(err, "kubeadm init error running command: %s", b.String())
	}

	//TODO(r2d4): get rid of global here
	master = k8s.NodeName
	if err := util.RetryAfter(100, unmarkMaster, time.Millisecond*500); err != nil {
		return errors.Wrap(err, "timed out waiting to unmark master")
	}

	if err := util.RetryAfter(100, elevateKubeSystemPrivileges, time.Millisecond*500); err != nil {
		return errors.Wrap(err, "timed out waiting to elevate kube-system RBAC privileges")
	}

	return nil
}

//TODO(r2d4): Split out into shared function between localkube and kubeadm
func addAddons(files *[]assets.CopyableFile) error {
	// add addons to file list
	// custom addons
	if err := assets.AddMinikubeDirAssets(files); err != nil {
		return errors.Wrap(err, "adding minikube dir assets")
	}
	// bundled addons
	for addonName, addonBundle := range assets.Addons {
		// TODO(r2d4): Kubeadm ignores the kube-dns addon and uses its own.
		// expose this in a better way
		if addonName == "kube-dns" {
			continue
		}
		if isEnabled, err := addonBundle.IsEnabled(); err == nil && isEnabled {
			for _, addon := range addonBundle.Assets {
				*files = append(*files, addon)
			}
		} else if err != nil {
			return nil
		}
	}

	return nil
}

func (k *KubeadmBootstrapper) RestartCluster(k8s bootstrapper.KubernetesConfig) error {
	opts := struct {
		KubeadmConfigFile string
	}{
		KubeadmConfigFile: constants.KubeadmConfigFile,
	}

	b := bytes.Buffer{}
	if err := kubeadmRestoreTemplate.Execute(&b, opts); err != nil {
		return err
	}

	if err := k.c.Run(b.String()); err != nil {
		return errors.Wrapf(err, "running cmd: %s", b.String())
	}

	if err := restartKubeProxy(k8s); err != nil {
		return errors.Wrap(err, "restarting kube-proxy")
	}

	return nil
}

func (k *KubeadmBootstrapper) SetupCerts(k8s bootstrapper.KubernetesConfig) error {
	return bootstrapper.SetupCerts(k.c, k8s)
}

// SetContainerRuntime possibly sets the container runtime, if it hasn't already
// been specified by the extra-config option.  It has a set of defaults known to
// work for a particular runtime.
func SetContainerRuntime(cfg map[string]string, runtime string) map[string]string {
	if _, ok := cfg["container-runtime"]; ok {
		glog.Infoln("Container runtime already set through extra options, ignoring --container-runtime flag.")
		return cfg
	}

	if runtime == "" {
		glog.Infoln("Container runtime flag provided with no value, using defaults.")
		return cfg
	}

	switch runtime {
	case "crio", "cri-o":
		cfg["container-runtime"] = "remote"
		cfg["container-runtime-endpoint"] = "/var/run/crio/crio.sock"
		cfg["image-service-endpoint"] = "/var/run/crio/crio.sock"
		cfg["runtime-request-timeout"] = "15m"
	default:
		cfg["container-runtime"] = runtime
	}

	return cfg
}

// NewKubeletConfig generates a new systemd unit containing a configured kubelet
// based on the options present in the KubernetesConfig.
func NewKubeletConfig(k8s bootstrapper.KubernetesConfig) (string, error) {
	version, err := ParseKubernetesVersion(k8s.KubernetesVersion)
	if err != nil {
		return "", errors.Wrap(err, "parsing kubernetes version")
	}

	extraOpts, err := ExtraConfigForComponent(Kubelet, k8s.ExtraOptions, version)
	if err != nil {
		return "", errors.Wrap(err, "generating extra configuration for kubelet")
	}

	extraOpts = SetContainerRuntime(extraOpts, k8s.ContainerRuntime)
	extraFlags := convertToFlags(extraOpts)
	b := bytes.Buffer{}
	opts := struct {
		ExtraOptions     string
		FeatureGates     string
		ContainerRuntime string
	}{
		ExtraOptions:     extraFlags,
		FeatureGates:     k8s.FeatureGates,
		ContainerRuntime: k8s.ContainerRuntime,
	}
	if err := kubeletSystemdTemplate.Execute(&b, opts); err != nil {
		return "", err
	}

	return b.String(), nil
}

func (k *KubeadmBootstrapper) UpdateCluster(cfg bootstrapper.KubernetesConfig) error {
	if cfg.ShouldLoadCachedImages {
		// Make best effort to load any cached images
		go machine.LoadImages(k.c, constants.GetKubeadmCachedImages(cfg.KubernetesVersion), constants.ImageCacheDir)
	}
	kubeadmCfg, err := generateConfig(cfg)
	if err != nil {
		return errors.Wrap(err, "generating kubeadm cfg")
	}

	kubeletCfg, err := NewKubeletConfig(cfg)
	if err != nil {
		return errors.Wrap(err, "generating kubelet config")
	}

	files := []assets.CopyableFile{
		assets.NewMemoryAssetTarget([]byte(kubeletService), constants.KubeletServiceFile, "0640"),
		assets.NewMemoryAssetTarget([]byte(kubeletCfg), constants.KubeletSystemdConfFile, "0640"),
		assets.NewMemoryAssetTarget([]byte(kubeadmCfg), constants.KubeadmConfigFile, "0640"),
	}

	var g errgroup.Group
	for _, bin := range []string{"kubelet", "kubeadm"} {
		bin := bin
		g.Go(func() error {
			path, err := maybeDownloadAndCache(bin, cfg.KubernetesVersion)
			if err != nil {
				return errors.Wrapf(err, "downloading %s", bin)
			}
			f, err := assets.NewFileAsset(path, "/usr/bin", bin, "0641")
			if err != nil {
				return errors.Wrap(err, "making new file asset")
			}
			if err := k.c.Copy(f); err != nil {
				return errors.Wrapf(err, "transferring kubeadm file: %+v", f)
			}
			return nil
		})
	}
	if err := g.Wait(); err != nil {
		return errors.Wrap(err, "downloading binaries")
	}

	if err := addAddons(&files); err != nil {
		return errors.Wrap(err, "adding addons to copyable files")
	}

	for _, f := range files {
		if err := k.c.Copy(f); err != nil {
			return errors.Wrapf(err, "transferring kubeadm file: %+v", f)
		}
	}

	err = k.c.Run(`
sudo systemctl daemon-reload &&
sudo systemctl enable kubelet &&
sudo systemctl start kubelet
`)
	if err != nil {
		return errors.Wrap(err, "starting kubelet")
	}

	return nil
}

func generateConfig(k8s bootstrapper.KubernetesConfig) (string, error) {
	version, err := ParseKubernetesVersion(k8s.KubernetesVersion)
	if err != nil {
		return "", errors.Wrap(err, "parsing kubernetes version")
	}

	// generates a map of component to extra args for apiserver, controller-manager, and scheduler
	extraComponentConfig, err := NewComponentExtraArgs(k8s.ExtraOptions, version, k8s.FeatureGates)
	if err != nil {
		return "", errors.Wrap(err, "generating extra component config for kubeadm")
	}

	opts := struct {
		CertDir           string
		ServiceCIDR       string
		AdvertiseAddress  string
		APIServerPort     int
		KubernetesVersion string
		EtcdDataDir       string
		NodeName          string
		ExtraArgs         []ComponentExtraArgs
	}{
		CertDir:           util.DefaultCertPath,
		ServiceCIDR:       util.DefaultServiceCIDR,
		AdvertiseAddress:  k8s.NodeIP,
		APIServerPort:     util.APIServerPort,
		KubernetesVersion: k8s.KubernetesVersion,
		EtcdDataDir:       "/data", //TODO(r2d4): change to something else persisted
		NodeName:          k8s.NodeName,
		ExtraArgs:         extraComponentConfig,
	}

	b := bytes.Buffer{}
	if err := kubeadmConfigTemplate.Execute(&b, opts); err != nil {
		return "", err
	}

	return b.String(), nil
}

func maybeDownloadAndCache(binary, version string) (string, error) {
	targetDir := constants.MakeMiniPath("cache", version)
	targetFilepath := path.Join(targetDir, binary)

	_, err := os.Stat(targetFilepath)
	// If it exists, do no verification and continue
	if err == nil {
		return targetFilepath, nil
	}
	if !os.IsNotExist(err) {
		return "", errors.Wrapf(err, "stat %s version %s at %s", binary, version, targetDir)
	}

	if err = os.MkdirAll(targetDir, 0777); err != nil {
		return "", errors.Wrapf(err, "mkdir %s", targetDir)
	}

	url := constants.GetKubernetesReleaseURL(binary, version)
	options := download.FileOptions{
		Mkdirs: download.MkdirAll,
	}

	options.Checksum = constants.GetKubernetesReleaseURLSha1(binary, version)
	options.ChecksumHash = crypto.SHA1

	fmt.Printf("Downloading %s %s\n", binary, version)
	if err := download.ToFile(url, targetFilepath, options); err != nil {
		return "", errors.Wrapf(err, "Error downloading %s %s", binary, version)
	}
	fmt.Printf("Finished Downloading %s %s\n", binary, version)

	return targetFilepath, nil
}


text:
"""
Copyright (c) 2014, 2015 Marshall Farrier
license http://opensource.org/licenses/MIT

@author: Marshall Farrier
@contact: marshalldfarrier@gmail.com
@since: 2014-11-02
@summary: Unit tests for data module
"""

from functools import partial
import unittest

import numpy as np
import pandas as pd

import pynance as pn

class TestData(unittest.TestCase):

    def setUp(self):
        session_dates = [
                '2014-01-06',
                '2014-01-07',
                '2014-01-08',
                '2014-01-09',
                '2014-01-10',
                '2014-01-13',
                '2014-01-14',
                '2014-01-15',
                '2014-01-16',
                '2014-01-17']
        self.equity_data = pd.DataFrame(np.arange(1., 21.).reshape((10, 2)), index=session_dates,
                columns=['Volume', 'Adj Close'])
        self.equity_data.index.name = 'Date'

    def test_center_df(self):
        features = pd.DataFrame(data=np.random.random((112, 3)), columns=['1', '2', '3'])
        centered_features, feat_means = pn.data.center(features)
        means_of_centered = np.mean(centered_features.values, axis=0)
        for i in range(len(means_of_centered)):
            self.assertAlmostEqual(means_of_centered[i], 0.)
        self.assertTrue(isinstance(centered_features, pd.DataFrame))
        self.assertTrue(isinstance(feat_means, pd.DataFrame))

    def test_center_partial_df(self):
        features = pd.DataFrame(data=np.ones((23, 4)), columns=map(str, range(4)), dtype='float64')
        features.iloc[:, 1:] = np.random.random((23, 3))
        centered_features, feat_means = pn.data.center(features.iloc[:, 1:], out=features.iloc[:, 1:])
        means_of_centered = np.mean(features.values, axis=0)
        self.assertAlmostEqual(means_of_centered[0], 1.)
        for i in range(1, len(means_of_centered)):
            self.assertAlmostEqual(means_of_centered[i], 0.)
        self.assertTrue(isinstance(centered_features, pd.DataFrame))
        self.assertTrue(isinstance(feat_means, pd.DataFrame))

    def test_center_ndarray(self):
        features = np.random.random((27, 9))
        centered_features, feat_means = pn.data.center(features)
        means_of_centered = np.mean(centered_features, axis=0)
        for i in range(len(means_of_centered)):
            self.assertAlmostEqual(means_of_centered[i], 0.)
        self.assertTrue(isinstance(centered_features, np.ndarray))
        self.assertTrue(isinstance(feat_means, np.ndarray))

    def test_center_ndarray_partial(self):
        features = np.ones((16, 5))
        features[:, :-1] = np.random.random((16, 4))
        centered_features, feat_means = pn.data.center(features[:, :-1],
                out=features[:, :-1])
        means_of_centered = np.mean(features, axis=0)
        for i in range(len(means_of_centered) - 1):
            self.assertAlmostEqual(means_of_centered[i], 0.)
        self.assertAlmostEqual(means_of_centered[-1], 1.)
        self.assertTrue(isinstance(centered_features, np.ndarray))
        self.assertTrue(isinstance(feat_means, np.ndarray))

    def test_normalize_df(self):
        features = pd.DataFrame(data=np.random.random((112, 3)), columns=['1', '2', '3'])
        centered_features, _ = pn.data.center(features)
        normalized_features, stds = pn.data.normalize(features)
        stds_of_normalized = np.std(normalized_features.values, axis=0)
        for i in range(len(stds_of_normalized)):
            self.assertAlmostEqual(stds_of_normalized[i], 1.)
        self.assertTrue(isinstance(normalized_features, pd.DataFrame))
        self.assertTrue(isinstance(stds, pd.DataFrame))

    def test_normalize_partial_df(self):
        features = pd.DataFrame(data=np.ones((23, 4)), columns=map(str, range(4)), dtype='float64')
        features.iloc[:, 1:] = np.random.random((23, 3))
        _, _ = pn.data.center(features.iloc[:, 1:], out=features.iloc[:, 1:])
        normalized_features, stds = pn.data.normalize(features.iloc[:, 1:], out=features.iloc[:, 1:])
        stds_of_normalized = np.std(features.values, axis=0)
        self.assertAlmostEqual(stds_of_normalized[0], 0.)
        for i in range(1, len(stds_of_normalized)):
            self.assertAlmostEqual(stds_of_normalized[i], 1.)
        self.assertTrue(isinstance(normalized_features, pd.DataFrame))
        self.assertTrue(isinstance(stds, pd.DataFrame))

    def test_normalize_ndarray(self):
        features = np.random.random((27, 9))
        centered_features, _ = pn.data.center(features)
        normalized_features, stds = pn.data.normalize(features)
        stds_of_normalized = np.std(normalized_features, axis=0)
        for i in range(len(stds_of_normalized)):
            self.assertAlmostEqual(stds_of_normalized[i], 1.)
        self.assertTrue(isinstance(normalized_features, np.ndarray))
        self.assertTrue(isinstance(stds, np.ndarray))

    def test_normalize_ndarray_partial(self):
        features = np.ones((16, 5))
        features[:, :-1] = np.random.random((16, 4))
        _, _ = pn.data.center(features[:, :-1], out=features[:, :-1])
        normalized_features, stds = pn.data.normalize(features[:, :-1], out=features[:, :-1])
        stds_of_normalized = np.std(features, axis=0)
        self.assertAlmostEqual(stds_of_normalized[-1], 0.)
        for i in range(len(stds_of_normalized) - 1):
            self.assertAlmostEqual(stds_of_normalized[i], 1.)
        self.assertTrue(isinstance(normalized_features, np.ndarray))
        self.assertTrue(isinstance(stds, np.ndarray))

    def test_transform_default(self):
        n_sessions = 3
        features = pn.featurize(self.equity_data, n_sessions)
        transformed = pn.data.transform(features)
        norms = np.linalg.norm(np.float64(transformed.values), axis=1)
        for i in range(len(norms)):
            self.assertAlmostEqual(norms[i], 1.0)

    def test_transform_rows_vector(self):
        n_sessions = 3
        norm = 3.14159
        features = pn.featurize(self.equity_data, n_sessions)
        transformed = pn.data.transform(features, method="vector", norm=norm)
        norms = np.linalg.norm(np.float64(transformed.values), axis=1)
        for i in range(len(norms)):
            self.assertAlmostEqual(norms[i], norm)

    def test_transform_rows_mean(self):
        n_sessions = 3
        norm = 10.0
        features = pn.featurize(self.equity_data, n_sessions)
        transformed = pn.data.transform(features, method="mean", norm=norm, axis=0)
        means = transformed.mean(axis=1)
        for i in range(len(means)):
            self.assertAlmostEqual(means.iloc[i], norm)

    def test_transform_rows_last(self):
        n_sessions = 3
        features = pn.featurize(self.equity_data, n_sessions)
        labels = pd.DataFrame(features.iloc[:, -1] * 0.5, index=features.index)
        transformed_features, transformed_labels = pn.data.transform(features, method="last", labels=labels)
        for i in range(len(transformed_features.index)):
            self.assertAlmostEqual(transformed_features.iloc[i, -1], 1.0)
        for i in range(len(transformed_labels.index)):
            self.assertAlmostEqual(transformed_labels.iloc[i, 0], 0.5)

    def test_transform_rows_first(self):
        n_sessions = 3
        features = pn.featurize(self.equity_data, n_sessions)
        # 2 columns of labels
        labels = pd.DataFrame(index=features.index, columns=['Label1', 'Label2'])
        labels['Label1'] = features.iloc[:, 0] * 0.5
        labels['Label2'] = features.iloc[:, 0] * 2.0
        transformed_features, transformed_labels = pn.data.transform(features, method="first", labels=labels)
        for i in range(len(transformed_features.index)):
            self.assertAlmostEqual(transformed_features.iloc[i, 0], 1.0)
        for i in range(len(transformed_labels.index)):
            self.assertAlmostEqual(transformed_labels.iloc[i, 0], 0.5)
            self.assertAlmostEqual(transformed_labels.iloc[i, 1], 2.0)

    def test_transform_cols_vector(self):
        norm = 3.14159
        transformed = pn.data.transform(self.equity_data, method="vector", norm=norm, axis=1)
        norms = np.linalg.norm(np.float64(transformed.values), axis=0)
        for i in range(len(norms)):
            self.assertAlmostEqual(norms[i], norm)

    def test_transform_cols_mean(self):
        norm = 10.0
        transformed = pn.data.transform(self.equity_data, method="mean", norm=norm, axis=1)
        means = transformed.mean(axis=0)
        for i in range(len(means)):
            self.assertAlmostEqual(means.iloc[i], norm)

    def test_transform_cols_last(self):
        transformed = pn.data.transform(self.equity_data, method="last", axis=1)
        for i in range(len(transformed.columns)):
            self.assertAlmostEqual(transformed.iloc[-1, i], 1.0)

    def test_transform_cols_first(self):
        transformed = pn.data.transform(self.equity_data, method="first", axis=1)
        for i in range(len(transformed.columns)):
            self.assertAlmostEqual(transformed.iloc[0, i], 1.0)

if __name__ == '__main__':
    unittest.main()


text:
using System;
using Microsoft.TeamFoundation.Migration.Toolkit.Linking;
using Microsoft.TeamFoundation.Migration.Toolkit.Services;

namespace Microsoft.TeamFoundation.Migration.Tfs2008WitAdapter.Linking
{
    [Serializable]
    public class WorkItemArtifactType : ArtifactType
    {
        private const string REFERENCE_NAME = "Microsoft.TeamFoundation.Migration.TFS.ArtifactType.WorkItem";
        private const string FRIENDLY_NAME = "Team Foundation Server Work Item artifact type";

        public WorkItemArtifactType()
            : base(REFERENCE_NAME, FRIENDLY_NAME, WellKnownContentType.WorkItem.ReferenceName)
        {}
    }
}


text:
import sys
import httplib
import json
import logging
from optparse import OptionParser
import time
import csv
import ecxclient.sdk.client as client

logger = logging.getLogger('logger')
logger.setLevel(logging.INFO)

parser = OptionParser()
parser.add_option("--user", dest="username", help="ECX Username")
parser.add_option("--pass", dest="password", help="ECX Password")
parser.add_option("--host", dest="host", help="ECX Host, (ex. https://172.20.58.10:8443)")
parser.add_option("--provname", dest="provname", help="Provider Name")
parser.add_option("--provhost", dest="provhost", help="Provider Host Address")
parser.add_option("--provsite", dest="provsite", help="Porivder Site")
parser.add_option("--provcred", dest="provcred", help="Provider Credentials (ECX object name)")
parser.add_option("--provtype", dest="provtype", help="Provider Type (oracle, sql, osvolume, saphana, cache)")
parser.add_option("--provst", dest="provst", help="Provider Server Type (physical or virtual)")
parser.add_option("--provport", dest="provport", help="Provider Port (optional)")
parser.add_option("--provos", dest="provos", help="Provider OS (required if filesystem/osvolume, windows, linux, aix)")
parser.add_option("--provvc", dest="provvc", help="Provider vCenter (required if Virtual)")
parser.add_option("--provcom", dest="provcom", help="Provider Comment (optional)")

(options, args) = parser.parse_args()

session = client.EcxSession(options.host, options.username, options.password)

def validate_input():

    if (options.username is None or options.password is None or options.host is None):
        print "ECX login information missing"
        sys.exit(2)

    if (options.provname is None):
        print "Provider name missing"
        sys.exit(2)

    if (options.provhost is None):
        print "Provider hostname or IP missing"
        sys.exit(2)

    if (options.provsite is None):
        print "Provider site missing"
        sys.exit(2)

    if (options.provcred is None):
        print "Provider credentials missing"
        sys.exit(2)

    if (options.provtype is None or options.provst is None):
        print "Provider type or server type is missing"
        sys.exit(2)

    if (options.provtype.upper() == "ORACLE" or options.provtype.upper() == "SQL"
        or options.provtype.upper() == "OSVOLUME" or options.provtype.upper == "SAPHANA"
        or options.provtype.upper() == "CACHE"):
        options.provtype = options.provtype.lower()
    else:
        print "Provider type is invalid (oracle, sql, osvolume, saphana or cache)"
        sys.exit(2)

    if (options.provst.upper() == "VIRTUAL" or options.provst.upper() == "PHYSICAL"):
        options.provst = options.provst.lower()
    else:
        print "Provider server type is invalid (virtual or physical)"
        sys.exit(2)

    if (options.provtype == "osvolume" and options.provos is None):
        print "Provider OS is required if type is osvolume"
        sys.exit(2)
    elif (options.provtype.upper() == "ORACLE" or options.provtype.upper() == "SAPHANA"
          or options.provtype.upper() == "CACHE"):
        options.provos = "linux"
    elif (options.provtype.upper() == "SQL"):
        options.provos = "windows"
    else:
        print "Provider OS is invalid (windows, linux or aix)"
        sys.exit(2)

    if (options.provst == "virtual" and options.provvc is None):
        print "Provider vCenter is required for virtual providers"
        sys.exit(2)
    return None

def prettyprint(indata):
    print json.dumps(indata, sort_keys=True,indent=4, separators=(',', ': '))

def find_credential():
    userlist = client.EcxAPI(session, 'identityuser').list()
    for user in userlist:
        if (user['name'].upper() == options.provcred.upper()):
            return {"href": user['links']['self']['href']}
    print "Provider credentials not found"
    sys.exit(2)

def find_vcenter():
    vcenterlist = client.EcxAPI(session, 'vsphere').list()
    for vcenter in vcenterlist:
        if (vcenter['name'].upper() == options.provvc.upper()):
            return vcenter['id']
    print "Provider vCenter not found"
    sys.exit(2)

def find_site_id_by_name():
    sitelist = client.EcxAPI(session, 'site').list()
    for site in sitelist:
        if (site['name'].upper() == options.provsite.upper()):
            return site['id']
    print "Provider site not found"
    sys.exit(2)

def build_provider():
    provider = {}
    provider['name'] = options.provname
    provider['siteId'] = find_site_id_by_name()
    provider['hostAddress'] = options.provhost
    provider['applicationType'] = options.provtype
    provider['serverType'] = options.provst

    if (options.provcom is not None):
        provider['comment'] = options.provcom
    else:
        provider['comment'] = ""

    if (options.provtype == "sql"):
        options.provos = "windows"

    if (options.provst == "virtual"):
        provider['vsphereId'] = find_vcenter()
    else:
        provider['vsphereId'] = ""

    if (options.provtype == "osvolume"):
        provider['osType'] = options.provos

    if (options.provport is not None):
        provider['portNumber'] = options.provport.int()
    elif (options.provos == "windows"):
        provider['portNumber'] = 5985
    elif (options.provtype == "osvolume"):
        provider['portNumber'] = 22
    else:
        provider['portNumber'] = None

    provider['osuser'] = find_credential()
    provider['appCredentials'] = []
    provider['addToCatJob'] = True
    provider['useKeyAuthentication'] = False
    return provider

def register_provider():
    provider = build_provider()
    try:
        resp = client.EcxAPI(session, 'appserver').post(data=provider)
        print "Provider " + options.provname + " registerd."
    except client.requests.exceptions.HTTPError as e:
        error = json.loads(e.response.content)
        print "Error registering provider: " + error['id']

validate_input()
session.login()
register_provider()
session.delete('endeavour/session/')


text:
<iframe id="patterns_iframe" src="#"></iframe>
<script>
document.getElementById('patterns_iframe').src="https://patterns.brl.nyc/public/people/new?referrer=" + window.location.href;
var jQueryScriptOutputted = false;
function initJQuery() {
  //if the jQuery object isn't available
  if (typeof(jQuery) === 'undefined') {
    if (! jQueryScriptOutputted) {
      //only output the script once..
      jQueryScriptOutputted = true;
      //output the script (load it from google api)
      var s = document.createElement('script');
       s.src ="//ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.js";
      document.head.appendChild(s);
    }
    setTimeout("initJQuery()", 50);
  } else {
    jQuery(function() {
      var s = document.createElement('script');
      s.src = "https://patterns.brl.nyc/js/seamless.parent.min.js";
      document.head.appendChild(s);
      setTimeout("initSeamless()", 50);
    });
  }
}
function initSeamless(){
  jQuery(function() {
    if (jQuery.isFunction(jQuery.fn.seamless)){
      var child = jQuery('#patterns_iframe').seamless();
      var cssSheets = jQuery.map(document.styleSheets,function(n,style){
        return n.href ?  n.href : null;
      });
      child.send({ stylesheets: cssSheets});
    }else{
      setTimeout("initSeamless()", 50);
    }
  });
}
initJQuery();
</script>


text:
'use strict';
var filesizegzip = require('filesizegzip');

module.exports = function(grunt) {

	grunt.registerMultiTask('filesizegzip', 'Output file size information', function() {

		var options = this.options({
			gzip: grunt.option('gzip') || true
		});

		this.filesSrc.forEach(function(filepath) {
			grunt.log.ok(filepath +': '+ filesizegzip(grunt.file.read(filepath), options.gzip));
		});

	});

};