text:
RubyRetriever
==============
[![Gem Version](https://badge.fury.io/rb/rubyretriever.svg)](http://badge.fury.io/rb/rubyretriever)  [![Build Status](https://travis-ci.org/joenorton/rubyretriever.svg?branch=master)](https://travis-ci.org/joenorton/rubyretriever)

By Joe Norton

RubyRetriever is a Web Crawler, Scraper & File Harvester. Available as a command-line executable and as a crawling framework.

RubyRetriever (RR) uses asynchronous HTTP requests via [Eventmachine](https://github.com/eventmachine/eventmachine) & [Synchrony](https://github.com/igrigorik/em-synchrony) to crawl webpages *very quickly*. RR also uses a Ruby implementation of the [bloomfilter](https://github.com/igrigorik/bloomfilter-rb) in order to keep track of pages it has already crawled in a memory efficient manner.

**v1.4.3 Update (3/24/2016)** - Fixes problem with file downloads that had query strings, the filename was being saved with the querystrings still attached. No more.

**v1.4.2 Update (3/24/2016)** - Fixes problem with named anchors (divs) being counted as links.

**v1.4.1 Update (3/24/2016)** - Update gemfile & external dependency versioning

**v1.4.0 Update (3/24/2016)** - Several bug fixes.

Mission
-------
RubyRetriever aims to be the best command-line crawling and scraping package written in Ruby and a replacement for paid software such as Screaming Frog SEO Spider.

Roadmap?
Not sure. Feel free to offer your thoughts.

Some Potential Ideas:
* 'freeroam mode' - to go on cruising the net endlessly in fileharvest mode
* 'dead-link finder' mode - collects links returning 404, or other error msgs
* 'validate robots.txt' mode - outputs the bot-exposed sitemap of your site
* more sophisticated SEO analysis? replace screaming frog? this would include checks for canonical URL, maybe some keyword density checks, content length checks, etc.

Features
--------
* Asynchronous HTTP Requests thru EM & Synchrony
* Bloom filter for tracking visited pages
* Supports HTTPS
* Follows 301 redirects (if to same host)
* 3 CLI modes
	* Sitemap - Find all links on a website, output a valid XML sitemap, or just a CSV
	* File Harvest - find all files linked to on a website, option to autodownload
	* SEO  - collect important SEO info from every page, output to a CSV (or STDOUT)
* Run a Custom Block on a Per-Page basis (PageIterator)

Use cases
---------
**As an Executable**
With a single command at the terminal, RR can:
1. Crawl your website and output a *valid XML sitemap* based on what it found.
2. Crawl a target website and *download all files of a given filetype*.
3. Crawl a target website, *collect important SEO information* such as page titles, meta descriptions and h1 tags, and write it to CSV.

**Used in Custom scripts**
As of version 1.3.0, with the PageIterator class you can pass a custom block that will get run against each page during a crawl, and collect the results in an array. This means you can define for yourself whatever it is you want to collect from each page during the crawl.

Help & Forks Welcome!

Getting started
-----------
Install the gem
```sh
$ gem install rubyretriever
```

Using the Executable
--------------------
 **Example: Sitemap mode**
```sh
$ rr --sitemap CSV --progress --limit 10 http://www.cnet.com
```
OR -- SAME COMMAND
```sh
$ rr -s csv -p -l 10 http://www.cnet.com
```

This would map http://www.cnet.com until it crawled a max of 10 pages, then write the results to a CSV named cnet. Optionally, you could also use the format XML and RR would output the same URL list into a valid XML sitemap that could be submitted to Google.

 **Example: File Harvesting mode**
```sh
$ rr --files txt --verbose --limit 1 http://textfiles.com/programming/
```
OR -- SAME COMMAND
```sh
$ rr -f txt -v -l 1 http://textfiles.com/programming/
```

This would crawl http://textfiles.com/programming/ looking for txt files for only a single page, then write out a list of filepaths to txt files to the terminal. Optionally, you could have the script autodownload all the files by adding the -a/--auto flag.

**Example: SEO mode**
```sh
$ rr --seo --progress --limit 10 --out cnet-seo http://www.cnet.com
```
OR -- SAME COMMAND
```sh
$ rr -e -p -l 10 -o cnet-seo http://www.cnet.com
```

This would go to http://www.cnet.com and crawl a max of 10 pages, during which it would collect the SEO fields on those pages - this currently means [url, page title, meta description, h1 text, h2 text]. It would then write the fields to a csv named cnet-seo.

command-line arguments
-----------------------
Usage: rr [MODE FLAG] [OPTIONS] Target_URL

Where MODE FLAG is required, and is either:
	-s, --sitemap FORMAT  (only accepts CSV or XML atm)
	-f, --files FILETYPE
	-e, --seo

and OPTIONS is the applicable:
    -o, --out FILENAME                  *Dump fetch data as CSV*
    -p, --progress						*Outputs a progressbar*
    -v, --verbose                       *Output more information*
    -l, --limit PAGE_LIMIT_#            *set a max on the total number of crawled pages*
    -h, --help                          *Display this screen*

Using as a Library (starting as of version 1.3.0)
------------------

If you want to collect something, other than that which the executable allows, on a 'per page' basis then you want to use the PageIterator class. Then you can run whatever block you want against each individual page's source code located during the crawl.

Sample Script using **PageIterator**
```ruby
require 'retriever'
opts = {
  'maxpages' => 1
}
t = Retriever::PageIterator.new('http://www.basecamp.com', opts) do |page|
  [page.url, page.title]
end
puts t.result.to_s
```

```sh
>> [["http://www.basecamp.com", "Basecamp is everyone's favorite project management app."]]
```
Available methods on the page iterator:
* **#url** - returns full URL of current page
* **#source** - returns raw page source code
* **#title** - returns html decoded verson of curent page title
* **#desc** - returns html decoded verson of curent page meta description
* **#h1**  - returns html decoded verson of current page's h1 tag
* **#h2**  - returns html decoded verson of current page's h2 tag
* **#links** - returns array of all links on the page
* **#parse_internal** - returns array of current page's internal (same host) links
* **#parse_internal_visitable** - returns #parse_internal plus added filtering of only links that are visitable
* **#parse_seo** - returns array of current page's html decoded title, desc, h1 and h2
* **#parse_files** - returns array of downloaded files of type supplied as RR options (fileharvest options)

Current Requirements
------------
em-synchrony
ruby-progressbar
bloomfilter-rb
addressable
htmlentities

License
-------
See included 'LICENSE' file. It's the MIT license.


text:
package org.apache.ambari.server.state;

import static org.easymock.EasyMock.createNiceMock;
import static org.easymock.EasyMock.createStrictMock;
import static org.easymock.EasyMock.expect;
import static org.easymock.EasyMock.replay;
import static org.easymock.EasyMock.verify;

import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javax.persistence.EntityManager;

import org.apache.ambari.server.AmbariException;
import org.apache.ambari.server.H2DatabaseCleaner;
import org.apache.ambari.server.actionmanager.RequestFactory;
import org.apache.ambari.server.api.services.AmbariMetaInfo;
import org.apache.ambari.server.controller.AmbariCustomCommandExecutionHelper;
import org.apache.ambari.server.controller.AmbariManagementController;
import org.apache.ambari.server.controller.ClusterRequest;
import org.apache.ambari.server.controller.ConfigurationRequest;
import org.apache.ambari.server.controller.spi.ClusterController;
import org.apache.ambari.server.orm.DBAccessor;
import org.apache.ambari.server.orm.GuiceJpaInitializer;
import org.apache.ambari.server.orm.InMemoryDefaultTestModule;
import org.apache.ambari.server.orm.dao.HostRoleCommandDAO;
import org.apache.ambari.server.security.SecurityHelper;
import org.apache.ambari.server.security.TestAuthenticationFactory;
import org.apache.ambari.server.stack.StackManagerFactory;
import org.apache.ambari.server.state.cluster.ClusterFactory;
import org.apache.ambari.server.state.configgroup.ConfigGroup;
import org.apache.ambari.server.state.configgroup.ConfigGroupFactory;
import org.apache.ambari.server.state.host.HostFactory;
import org.apache.ambari.server.state.stack.OsFamily;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.security.core.context.SecurityContextHolder;

import com.google.inject.AbstractModule;
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.persist.Transactional;

import junit.framework.Assert;

@RunWith(Enclosed.class)
public class ConfigHelperTest {
  public static class RunWithInMemoryDefaultTestModule {
    private final static Logger LOG = LoggerFactory.getLogger(ConfigHelperTest.class);
    private static Clusters clusters;
    private static Injector injector;
    private static String clusterName;
    private static Cluster cluster;
    private static ConfigGroupFactory configGroupFactory;
    private static ConfigHelper configHelper;
    private static AmbariManagementController managementController;
    private static AmbariMetaInfo metaInfo;
    private static ConfigFactory configFactory;

    @Before
    public void setup() throws Exception {
      // Set the authenticated user
      // TODO: remove this or replace the authenticated user to test authorization rules
      SecurityContextHolder.getContext().setAuthentication(TestAuthenticationFactory.createAdministrator("admin"));

      injector = Guice.createInjector(new InMemoryDefaultTestModule());
      injector.getInstance(GuiceJpaInitializer.class);
      clusters = injector.getInstance(Clusters.class);
      configGroupFactory = injector.getInstance(ConfigGroupFactory.class);
      configHelper = injector.getInstance(ConfigHelper.class);
      managementController = injector.getInstance(AmbariManagementController.class);
      metaInfo = injector.getInstance(AmbariMetaInfo.class);
      configFactory = injector.getInstance(ConfigFactory.class);

      clusterName = "c1";
      clusters.addCluster(clusterName, new StackId("HDP-2.0.6"));
      cluster = clusters.getCluster(clusterName);
      Assert.assertNotNull(cluster);
      clusters.addHost("h1");
      clusters.addHost("h2");
      clusters.addHost("h3");
      Assert.assertNotNull(clusters.getHost("h1"));
      Assert.assertNotNull(clusters.getHost("h2"));
      Assert.assertNotNull(clusters.getHost("h3"));

      // core-site
      ConfigurationRequest cr = new ConfigurationRequest();
      cr.setClusterName(clusterName);
      cr.setType("core-site");
      cr.setVersionTag("version1");
      cr.setProperties(new HashMap<String, String>() {{
        put("ipc.client.connect.max.retries", "30");
        put("fs.trash.interval", "30");
      }});
      cr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("ipc.client.connect.max.retries", "1");
        attrs.put("fs.trash.interval", "2");
        put("attribute1", attrs);
      }});

      final ClusterRequest clusterRequest1 =
          new ClusterRequest(cluster.getClusterId(), clusterName,
              cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest1.setDesiredConfig(Collections.singletonList(cr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest1);
      }}, null);

      // flume-conf

      ConfigurationRequest cr2 = new ConfigurationRequest();
      cr2.setClusterName(clusterName);
      cr2.setType("flume-conf");
      cr2.setVersionTag("version1");

      final ClusterRequest clusterRequest2 =
          new ClusterRequest(cluster.getClusterId(), clusterName,
              cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest2.setDesiredConfig(Collections.singletonList(cr2));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest2);
      }}, null);

      // global
      cr.setType("global");
      cr.setVersionTag("version1");
      cr.setProperties(new HashMap<String, String>() {{
        put("dfs_namenode_name_dir", "/hadoop/hdfs/namenode");
        put("namenode_heapsize", "1024");
      }});
      cr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("dfs_namenode_name_dir", "3");
        attrs.put("namenode_heapsize", "4");
        put("attribute2", attrs);
      }});

      final ClusterRequest clusterRequest3 =
          new ClusterRequest(cluster.getClusterId(), clusterName,
              cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest3.setDesiredConfig(Collections.singletonList(cr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest3);
      }}, null);

      // oozie-site
      ConfigurationRequest cr4 = new ConfigurationRequest();
      cr4.setClusterName(clusterName);
      cr4.setType("oozie-site");
      cr4.setVersionTag("version1");
      cr4.setProperties(new HashMap<String, String>() {{
        put("oozie.authentication.type", "simple");
        put("oozie.service.HadoopAccessorService.kerberos.enabled", "false");
      }});
      cr4.setPropertiesAttributes(null);

      final ClusterRequest clusterRequest4 =
          new ClusterRequest(cluster.getClusterId(), clusterName,
              cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest4.setDesiredConfig(Collections.singletonList(cr4));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest4);
      }}, null);

      // ams-site
      ConfigurationRequest cr5 = new ConfigurationRequest();
      cr5.setClusterName(clusterName);
      cr5.setType("ams-site");
      cr5.setVersionTag("version1");
      cr5.setProperties(new HashMap<String, String>() {{
        put("timeline.service.operating.mode", "embedded");
        put("timeline.service.fifo.enabled", "false");
      }});
      cr5.setPropertiesAttributes(null);

      final ClusterRequest clusterRequest5 =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest5.setDesiredConfig(Collections.singletonList(cr5));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest5);
      }}, null);
    }

    @After
    public void tearDown() throws AmbariException, SQLException {
      H2DatabaseCleaner.clearDatabaseAndStopPersistenceService(injector);

      // Clear the authenticated user
      SecurityContextHolder.getContext().setAuthentication(null);
    }

    @Transactional
    Long addConfigGroup(String name, String tag, List<String> hosts,
                        List<Config> configs) throws AmbariException {

      Map<Long, Host> hostMap = new HashMap<>();
      Map<String, Config> configMap = new HashMap<>();

      Long hostId = 1L;
      for (String hostname : hosts) {
        Host host = clusters.getHost(hostname);
        hostMap.put(hostId, host);
        hostId++;
      }

      for (Config config : configs) {
        configMap.put(config.getType(), config);
      }

      ConfigGroup configGroup = configGroupFactory.createNew(cluster, name,
          tag, "", configMap, hostMap);
      LOG.info("Config group created with tag " + tag);
      configGroup.setTag(tag);

      cluster.addConfigGroup(configGroup);

      return configGroup.getId();
    }

    void applyConfig(Map<String, String> properties, String configType, String configTag) throws Exception {
      ConfigurationRequest cr = new ConfigurationRequest();
      cr.setClusterName(clusterName);
      cr.setType(configType);
      cr.setVersionTag(configTag);
      cr.setProperties(properties);

      final ClusterRequest clusterRequest =
          new ClusterRequest(cluster.getClusterId(), clusterName,
              cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest.setDesiredConfig(Collections.singletonList(cr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest);
      }}, null);
    }

    @Test
    public void testProcessHiddenAttribute() throws Exception {
      StackInfo stackInfo = metaInfo.getStack("HDP", "2.0.5");
      Map<String, Map<String, Map<String, String>>> configAttributes = new HashMap<>();
      configAttributes.put("hive-site", stackInfo.getDefaultConfigAttributesForConfigType("hive-site"));

      Map<String, Map<String, String>> originalConfig_hiveClient = createHiveConfig();

      Map<String, Map<String, String>> expectedConfig_hiveClient = new HashMap<String, Map<String, String>>() {{
        put("hive-site", new HashMap<String, String>() {{
          put("javax.jdo.option.ConnectionDriverName", "oracle");
          put("hive.metastore.warehouse.dir", "/tmp");
        }});
      }};

      ConfigHelper.processHiddenAttribute(originalConfig_hiveClient, configAttributes, "HIVE_CLIENT", false);
      Assert.assertEquals(expectedConfig_hiveClient, originalConfig_hiveClient);

      Map<String, Map<String, String>> originalConfig_hiveServer = createHiveConfig();
      Map<String, Map<String, String>> expectedConfig_hiveServer = createHiveConfig();

      ConfigHelper.processHiddenAttribute(originalConfig_hiveServer, configAttributes, "HIVE_SERVER", false);
      Assert.assertEquals(expectedConfig_hiveServer, originalConfig_hiveServer);

      Map<String, Map<String, String>> originalConfig_hiveServer1 = createHiveConfig();
      Map<String, Map<String, String>> expectedConfig_hiveServer1 = expectedConfig_hiveClient;

      // config download removes hidden properties without respecting of component
      ConfigHelper.processHiddenAttribute(originalConfig_hiveServer1, configAttributes, "HIVE_SERVER", true);
      Assert.assertEquals(expectedConfig_hiveServer1, originalConfig_hiveServer1);
    }

    private Map<String, Map<String, String>> createHiveConfig() {
      return new HashMap<String, Map<String, String>>() {{
        put("hive-site", new HashMap<String, String>() {{
          put("javax.jdo.option.ConnectionDriverName", "oracle");
          put("javax.jdo.option.ConnectionPassword", "1");
          put("hive.metastore.warehouse.dir", "/tmp");
        }});
      }};
    }

    @Test
    public void testEffectiveTagsForHost() throws Exception {

      //Setup
      ConfigurationRequest cr5 = new ConfigurationRequest();
      cr5.setClusterName(clusterName);
      cr5.setType("ams-env");
      cr5.setVersionTag("version1");
      cr5.setProperties(new HashMap<String, String>() {{
        put("metrics_collector_log_dir", "/var/log/ambari-metrics-collector");
        put("metrics_collector_pid_dir", "/var/run/ambari-metrics-collector");
      }});
      cr5.setPropertiesAttributes(null);

      final ClusterRequest clusterRequest6 =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest6.setDesiredConfig(Collections.singletonList(cr5));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest6);
      }}, null);

      Map<String, String> properties = new HashMap<>();
      properties.put("a", "b");
      properties.put("c", "d");

      final Config config = configFactory.createNew(cluster, "ams-env", "version122", properties, null);
      Long groupId = addConfigGroup("g1", "t1", new ArrayList<String>() {{
        add("h1");
      }}, new ArrayList<Config>() {{
        add(config);
      }});

      Assert.assertNotNull(groupId);

      Map<String, Map<String, String>> configTags = configHelper
          .getEffectiveDesiredTags(cluster, "h1");

      Assert.assertNotNull(configTags);
      Map<String, String> tagsWithOverrides = configTags.get("ams-env");
      Assert.assertNotNull(tagsWithOverrides);
      Assert.assertTrue(tagsWithOverrides.containsKey(ConfigHelper.CLUSTER_DEFAULT_TAG));
      Assert.assertEquals("version1", tagsWithOverrides.get(ConfigHelper.CLUSTER_DEFAULT_TAG));
      Assert.assertTrue(tagsWithOverrides.containsKey(groupId.toString()));
      Assert.assertEquals("version122", tagsWithOverrides.get(groupId.toString()));
    }

    @Test
    public void testEffectivePropertiesWithOverrides() throws Exception {

      //Setup
      // core-site
      ConfigurationRequest cr = new ConfigurationRequest();
      cr.setClusterName(clusterName);
      cr.setType("core-site2");
      cr.setVersionTag("version1");
      cr.setProperties(new HashMap<String, String>() {{
        put("ipc.client.connect.max.retries", "30");
        put("fs.trash.interval", "30");
      }});
      cr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("ipc.client.connect.max.retries", "1");
        attrs.put("fs.trash.interval", "2");
        put("attribute1", attrs);
      }});

      final ClusterRequest clusterRequest1 =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest1.setDesiredConfig(Collections.singletonList(cr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest1);
      }}, null);

      // global
      cr.setType("global2");
      cr.setVersionTag("version1");
      cr.setProperties(new HashMap<String, String>() {{
        put("dfs_namenode_name_dir", "/hadoop/hdfs/namenode");
        put("namenode_heapsize", "1024");
      }});
      cr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("dfs_namenode_name_dir", "3");
        attrs.put("namenode_heapsize", "4");
        put("attribute2", attrs);
      }});

      final ClusterRequest clusterRequest3 =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequest3.setDesiredConfig(Collections.singletonList(cr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequest3);
      }}, null);

      Map<String, String> properties = new HashMap<>();
      properties.put("a", "b");
      properties.put("c", "d");
      final Config config1 = configFactory.createNew(cluster, "core-site2", "version122", properties, null);

      Map<String, String> properties2 = new HashMap<>();
      properties2.put("namenode_heapsize", "1111");
      final Config config2 = configFactory.createNew(cluster, "global2", "version122", properties2, null);

      Long groupId = addConfigGroup("g2", "t1", new ArrayList<String>() {{
        add("h1");
      }}, new ArrayList<Config>() {{
        add(config1);
        add(config2);
      }});

      Assert.assertNotNull(groupId);

      Map<String, Map<String, String>> propertyMap = configHelper
          .getEffectiveConfigProperties(cluster,
              configHelper.getEffectiveDesiredTags(cluster, "h1"));

      Assert.assertNotNull(propertyMap);
      Assert.assertTrue(propertyMap.containsKey("global2"));
      Map<String, String> globalProps = propertyMap.get("global2");
      Assert.assertEquals("1111", globalProps.get("namenode_heapsize"));
      Assert.assertEquals("/hadoop/hdfs/namenode", globalProps.get("dfs_namenode_name_dir"));
      Assert.assertTrue(propertyMap.containsKey("core-site"));
      Map<String, String> coreProps = propertyMap.get("core-site2");
      Assert.assertTrue(coreProps.containsKey("a"));
      Assert.assertTrue(coreProps.containsKey("c"));
      Assert.assertEquals("30", coreProps.get("ipc.client.connect.max.retries"));
    }

    @Test
    public void testEffectivePropertiesAttributesWithOverrides() throws Exception {

      //Another version of core-site & global.
      // core-site3
      ConfigurationRequest crr = new ConfigurationRequest();
      crr.setClusterName(clusterName);
      crr.setType("core-site3");
      crr.setVersionTag("version1");
      crr.setProperties(new HashMap<String, String>() {{
        put("ipc.client.connect.max.retries", "30");
        put("fs.trash.interval", "30");
      }});
      crr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("ipc.client.connect.max.retries", "1");
        attrs.put("fs.trash.interval", "2");
        put("attribute1", attrs);
      }});

      final ClusterRequest clusterRequestDup =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequestDup.setDesiredConfig(Collections.singletonList(crr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequestDup);
      }}, null);

      // global3
      crr.setType("global3");
      crr.setVersionTag("version1");
      crr.setProperties(new HashMap<String, String>() {{
        put("dfs_namenode_name_dir", "/hadoop/hdfs/namenode");
        put("namenode_heapsize", "1024");
      }});
      crr.setPropertiesAttributes(new HashMap<String, Map<String, String>>() {{
        Map<String, String> attrs = new HashMap<>();
        attrs.put("dfs_namenode_name_dir", "3");
        attrs.put("namenode_heapsize", "4");
        put("attribute2", attrs);
      }});

      final ClusterRequest clusterRequestGlobalDup =
        new ClusterRequest(cluster.getClusterId(), clusterName,
          cluster.getDesiredStackVersion().getStackVersion(), null);

      clusterRequestGlobalDup.setDesiredConfig(Collections.singletonList(crr));
      managementController.updateClusters(new HashSet<ClusterRequest>() {{
        add(clusterRequestGlobalDup);
      }}, null);

      Map<String, String> attributes = new HashMap<>();
      attributes.put("fs.trash.interval", "11");
      attributes.put("b", "y");
      Map<String, Map<String, String>> config1Attributes = new HashMap<>();
      config1Attributes.put("attribute1", attributes);

      final Config config1 = configFactory.createNew(cluster, "core-site3", "version122",
          new HashMap<String, String>(), config1Attributes);

      attributes = new HashMap<>();
      attributes.put("namenode_heapsize", "z");
      attributes.put("c", "q");
      Map<String, Map<String, String>> config2Attributes = new HashMap<>();
      config2Attributes.put("attribute2", attributes);

      final Config config2 = configFactory.createNew(cluster, "global3", "version122",
          new HashMap<String, String>(), config2Attributes);

      Long groupId = addConfigGroup("g3", "t1", new ArrayList<String>() {{
        add("h3");
      }}, new ArrayList<Config>() {{
        add(config1);
        add(config2);
      }});

      Assert.assertNotNull(groupId);

      Map<String, Map<String, Map<String, String>>> effectiveAttributes = configHelper
          .getEffectiveConfigAttributes(cluster,
              configHelper.getEffectiveDesiredTags(cluster, "h3"));

      Assert.assertNotNull(effectiveAttributes);
      Assert.assertEquals(7, effectiveAttributes.size());

      Assert.assertTrue(effectiveAttributes.containsKey("global3"));
      Map<String, Map<String, String>> globalAttrs = effectiveAttributes.get("global3");
      Assert.assertEquals(1, globalAttrs.size());
      Assert.assertTrue(globalAttrs.containsKey("attribute2"));
      Map<String, String> attribute2Occurances = globalAttrs.get("attribute2");
      Assert.assertEquals(3, attribute2Occurances.size());
      Assert.assertTrue(attribute2Occurances.containsKey("namenode_heapsize"));
      Assert.assertEquals("z", attribute2Occurances.get("namenode_heapsize"));
      Assert.assertTrue(attribute2Occurances.containsKey("dfs_namenode_name_dir"));
      Assert.assertEquals("3", attribute2Occurances.get("dfs_namenode_name_dir"));
      Assert.assertTrue(attribute2Occurances.containsKey("c"));
      Assert.assertEquals("q", attribute2Occurances.get("c"));

      Assert.assertTrue(effectiveAttributes.containsKey("core-site3"));
      Map<String, Map<String, String>> coreAttrs = effectiveAttributes.get("core-site3");
      Assert.assertEquals(1, coreAttrs.size());
      Assert.assertTrue(coreAttrs.containsKey("attribute1"));
      Map<String, String> attribute1Occurances = coreAttrs.get("attribute1");
      Assert.assertEquals(3, attribute1Occurances.size());
      Assert.assertTrue(attribute1Occurances.containsKey("ipc.client.connect.max.retries"));
      Assert.assertEquals("1", attribute1Occurances.get("ipc.client.connect.max.retries"));
      Assert.assertTrue(attribute1Occurances.containsKey("fs.trash.interval"));
      Assert.assertEquals("11", attribute1Occurances.get("fs.trash.interval"));
      Assert.assertTrue(attribute1Occurances.containsKey("b"));
      Assert.assertEquals("y", attribute1Occurances.get("b"));
    }

    @Test
    public void testCloneAttributesMap() throws Exception {
      // init
      Map<String, Map<String, String>> targetAttributesMap = new HashMap<>();
      Map<String, String> attributesValues = new HashMap<>();
      attributesValues.put("a", "1");
      attributesValues.put("b", "2");
      attributesValues.put("f", "3");
      attributesValues.put("q", "4");
      targetAttributesMap.put("attr", attributesValues);
      Map<String, Map<String, String>> sourceAttributesMap = new HashMap<>();
      attributesValues = new HashMap<>();
      attributesValues.put("a", "5");
      attributesValues.put("f", "6");
      sourceAttributesMap.put("attr", attributesValues);
      attributesValues = new HashMap<>();
      attributesValues.put("f", "7");
      attributesValues.put("q", "8");
      sourceAttributesMap.put("attr1", attributesValues);

      // eval
      configHelper.cloneAttributesMap(sourceAttributesMap, targetAttributesMap);

      // verification
      Assert.assertEquals(2, targetAttributesMap.size());
      Assert.assertTrue(targetAttributesMap.containsKey("attr"));
      Assert.assertTrue(targetAttributesMap.containsKey("attr1"));
      Map<String, String> attributes = targetAttributesMap.get("attr");
      Assert.assertEquals(4, attributes.size());
      Assert.assertEquals("5", attributes.get("a"));
      Assert.assertEquals("2", attributes.get("b"));
      Assert.assertEquals("6", attributes.get("f"));
      Assert.assertEquals("4", attributes.get("q"));
      attributes = targetAttributesMap.get("attr1");
      Assert.assertEquals(2, attributes.size());
      Assert.assertEquals("7", attributes.get("f"));
      Assert.assertEquals("8", attributes.get("q"));
    }

    @Test
    public void testCloneAttributesMapSourceIsNull() throws Exception {
      // init
      Map<String, Map<String, String>> targetAttributesMap = new HashMap<>();
      Map<String, String> attributesValues = new HashMap<>();
      attributesValues.put("a", "1");
      attributesValues.put("b", "2");
      attributesValues.put("f", "3");
      attributesValues.put("q", "4");
      targetAttributesMap.put("attr", attributesValues);
      Map<String, Map<String, String>> sourceAttributesMap = null;

      // eval
      configHelper.cloneAttributesMap(sourceAttributesMap, targetAttributesMap);

      // verification
      // No exception should be thrown
      // targetMap should not be changed
      Assert.assertEquals(1, targetAttributesMap.size());
      Assert.assertTrue(targetAttributesMap.containsKey("attr"));
      Map<String, String> attributes = targetAttributesMap.get("attr");
      Assert.assertEquals(4, attributes.size());
      Assert.assertEquals("1", attributes.get("a"));
      Assert.assertEquals("2", attributes.get("b"));
      Assert.assertEquals("3", attributes.get("f"));
      Assert.assertEquals("4", attributes.get("q"));
    }

    @Test
    public void testCloneAttributesMapTargetIsNull() throws Exception {
      // init
      Map<String, Map<String, String>> targetAttributesMap = null;
      Map<String, Map<String, String>> sourceAttributesMap = new HashMap<>();
      Map<String, String> attributesValues = new HashMap<>();
      attributesValues.put("a", "5");
      attributesValues.put("f", "6");
      sourceAttributesMap.put("attr", attributesValues);
      attributesValues = new HashMap<>();
      attributesValues.put("f", "7");
      attributesValues.put("q", "8");
      sourceAttributesMap.put("attr1", attributesValues);

      // eval
      configHelper.cloneAttributesMap(sourceAttributesMap, targetAttributesMap);

      // verification
      // No exception should be thrown
      // sourceMap should not be changed
      Assert.assertEquals(2, sourceAttributesMap.size());
      Assert.assertTrue(sourceAttributesMap.containsKey("attr"));
      Assert.assertTrue(sourceAttributesMap.containsKey("attr1"));
      Map<String, String> attributes = sourceAttributesMap.get("attr");
      Assert.assertEquals(2, attributes.size());
      Assert.assertEquals("5", attributes.get("a"));
      Assert.assertEquals("6", attributes.get("f"));
      attributes = sourceAttributesMap.get("attr1");
      Assert.assertEquals(2, attributes.size());
      Assert.assertEquals("7", attributes.get("f"));
      Assert.assertEquals("8", attributes.get("q"));
    }

    @Test
    public void testMergeAttributes() throws Exception {
      Map<String, Map<String, String>> persistedAttributes = new HashMap<>();
      Map<String, String> persistedFinalAttrs = new HashMap<>();
      persistedFinalAttrs.put("a", "true");
      persistedFinalAttrs.put("c", "true");
      persistedFinalAttrs.put("d", "true");
      persistedAttributes.put("final", persistedFinalAttrs);
      Map<String, Map<String, String>> confGroupAttributes = new HashMap<>();
      Map<String, String> confGroupFinalAttrs = new HashMap<>();
      confGroupFinalAttrs.put("b", "true");
      confGroupAttributes.put("final", confGroupFinalAttrs);
      Map<String, String> confGroupProperties = new HashMap<>();
      confGroupProperties.put("a", "any");
      confGroupProperties.put("b", "any");
      confGroupProperties.put("c", "any");

      Config overrideConfig = configFactory.createNew(cluster, "type", null,
          confGroupProperties, confGroupAttributes);

      Map<String, Map<String, String>> result
          = configHelper.overrideAttributes(overrideConfig, persistedAttributes);

      Assert.assertNotNull(result);
      Assert.assertEquals(1, result.size());
      Map<String, String> finalResultAttributes = result.get("final");
      Assert.assertNotNull(finalResultAttributes);
      Assert.assertEquals(2, finalResultAttributes.size());
      Assert.assertEquals("true", finalResultAttributes.get("b"));
      Assert.assertEquals("true", finalResultAttributes.get("d"));
    }

    @Test
    public void testMergeAttributesWithNoAttributeOverrides() throws Exception {
      Map<String, Map<String, String>> persistedAttributes = new HashMap<>();
      Map<String, String> persistedFinalAttrs = new HashMap<>();
      persistedFinalAttrs.put("a", "true");
      persistedFinalAttrs.put("c", "true");
      persistedFinalAttrs.put("d", "true");
      persistedAttributes.put("final", persistedFinalAttrs);
      Map<String, Map<String, String>> confGroupAttributes = new HashMap<>();
      Map<String, String> confGroupProperties = new HashMap<>();
      confGroupProperties.put("a", "any");
      confGroupProperties.put("b", "any");
      confGroupProperties.put("c", "any");

      Config overrideConfig = configFactory.createNew(cluster, "type", null,
          confGroupProperties, confGroupAttributes);

      Map<String, Map<String, String>> result
          = configHelper.overrideAttributes(overrideConfig, persistedAttributes);

      Assert.assertNotNull(result);
      Assert.assertEquals(1, result.size());
      Map<String, String> finalResultAttributes = result.get("final");
      Assert.assertNotNull(finalResultAttributes);
      Assert.assertEquals(1, finalResultAttributes.size());
      Assert.assertEquals("true", finalResultAttributes.get("d"));
    }

    @Test
    public void testMergeAttributesWithNullAttributes() throws Exception {
      Map<String, Map<String, String>> persistedAttributes = new HashMap<>();
      Map<String, String> persistedFinalAttrs = new HashMap<>();
      persistedFinalAttrs.put("a", "true");
      persistedFinalAttrs.put("c", "true");
      persistedFinalAttrs.put("d", "true");
      persistedAttributes.put("final", persistedFinalAttrs);
      Map<String, String> confGroupProperties = new HashMap<>();
      confGroupProperties.put("a", "any");
      confGroupProperties.put("b", "any");
      confGroupProperties.put("c", "any");

      Config overrideConfig = configFactory.createNew(cluster, "type", null,
          confGroupProperties, null);

      Map<String, Map<String, String>> result
          = configHelper.overrideAttributes(overrideConfig, persistedAttributes);

      Assert.assertNotNull(result);
      Assert.assertEquals(1, result.size());
      Map<String, String> finalResultAttributes = result.get("final");
      Assert.assertNotNull(finalResultAttributes);
      Assert.assertEquals(3, finalResultAttributes.size());
      Assert.assertEquals("true", finalResultAttributes.get("a"));
      Assert.assertEquals("true", finalResultAttributes.get("c"));
      Assert.assertEquals("true", finalResultAttributes.get("d"));
    }

    @Test
    public void testFilterInvalidPropertyValues() {
      Map<PropertyInfo, String> properties = new HashMap<>();
      PropertyInfo prop1 = new PropertyInfo();
      prop1.setName("1");
      PropertyInfo prop2 = new PropertyInfo();
      prop1.setName("2");
      PropertyInfo prop3 = new PropertyInfo();
      prop1.setName("3");
      PropertyInfo prop4 = new PropertyInfo();
      prop1.setName("4");

      properties.put(prop1, "/tmp");
      properties.put(prop2, "null");
      properties.put(prop3, "");
      properties.put(prop4, null);

      Set<String> resultSet = configHelper.filterInvalidPropertyValues(properties, "testlist");
      Assert.assertEquals(1, resultSet.size());
      Assert.assertEquals(resultSet.iterator().next(), "/tmp");
    }

    @Test
    public void testMergeAttributesWithNullProperties() throws Exception {
      Map<String, Map<String, String>> persistedAttributes = new HashMap<>();
      Map<String, String> persistedFinalAttrs = new HashMap<>();
      persistedFinalAttrs.put("a", "true");
      persistedFinalAttrs.put("c", "true");
      persistedFinalAttrs.put("d", "true");
      persistedAttributes.put("final", persistedFinalAttrs);
      Map<String, Map<String, String>> confGroupAttributes = new HashMap<>();
      Map<String, String> confGroupFinalAttrs = new HashMap<>();
      confGroupFinalAttrs.put("b", "true");
      confGroupAttributes.put("final", confGroupFinalAttrs);

      Config overrideConfig = configFactory.createNew(cluster, "type", "version122",
          new HashMap<String,String>(), confGroupAttributes);

      Map<String, Map<String, String>> result
          = configHelper.overrideAttributes(overrideConfig, persistedAttributes);

      Assert.assertNotNull(result);
      Assert.assertEquals(1, result.size());
      Map<String, String> finalResultAttributes = result.get("final");
      Assert.assertNotNull(finalResultAttributes);
      Assert.assertEquals(4, finalResultAttributes.size());
      Assert.assertEquals("true", finalResultAttributes.get("a"));
      Assert.assertEquals("true", finalResultAttributes.get("b"));
      Assert.assertEquals("true", finalResultAttributes.get("c"));
      Assert.assertEquals("true", finalResultAttributes.get("d"));
    }

    @Test
    public void testUpdateConfigType() throws Exception {
      Config currentConfig = cluster.getDesiredConfigByType("core-site");
      Map<String, String> properties = currentConfig.getProperties();
      // Attributes exist
      Map<String, Map<String, String>> propertiesAttributes = currentConfig.getPropertiesAttributes();
      Assert.assertNotNull(propertiesAttributes);
      Assert.assertEquals(1, propertiesAttributes.size());
      Assert.assertTrue(propertiesAttributes.containsKey("attribute1"));
      // Config tag before update
      Assert.assertEquals("version1", currentConfig.getTag());
      // Properties before update
      Assert.assertEquals("30", properties.get("fs.trash.interval"));
      // Property and attribute exist
      Assert.assertTrue(properties.containsKey("ipc.client.connect.max.retries"));
      Assert.assertTrue(propertiesAttributes.get("attribute1").containsKey("ipc.client.connect.max.retries"));

      Map<String, String> updates = new HashMap<>();
      updates.put("new-property", "new-value");
      updates.put("fs.trash.interval", "updated-value");
      Collection<String> removals = Collections.singletonList("ipc.client.connect.max.retries");
      configHelper.updateConfigType(cluster, managementController, "core-site", updates, removals, "admin", "Test note");

      Config updatedConfig = cluster.getDesiredConfigByType("core-site");
      // Attributes aren't lost
      propertiesAttributes = updatedConfig.getPropertiesAttributes();
      Assert.assertNotNull(propertiesAttributes);
      Assert.assertEquals(1, propertiesAttributes.size());
      Assert.assertTrue(propertiesAttributes.containsKey("attribute1"));
      // Config tag updated
      Assert.assertFalse("version1".equals(updatedConfig.getTag()));
      // Property added
      properties = updatedConfig.getProperties();
      Assert.assertTrue(properties.containsKey("new-property"));
      Assert.assertEquals("new-value", properties.get("new-property"));
      // Property updated
      Assert.assertTrue(properties.containsKey("fs.trash.interval"));
      Assert.assertEquals("updated-value", properties.get("fs.trash.interval"));
      Assert.assertEquals("2", propertiesAttributes.get("attribute1").get("fs.trash.interval"));
      // Property and attribute removed
      Assert.assertFalse(properties.containsKey("ipc.client.connect.max.retries"));
      Assert.assertFalse(propertiesAttributes.get("attribute1").containsKey("ipc.client.connect.max.retries"));
    }

    @Test
    public void testUpdateConfigTypeNoPropertyAttributes() throws Exception {
      Config currentConfig = cluster.getDesiredConfigByType("oozie-site");
      Map<String, String> properties = currentConfig.getProperties();
      // Config tag before update
      Assert.assertEquals("version1", currentConfig.getTag());
      // Properties before update
      Assert.assertEquals("simple", properties.get("oozie.authentication.type"));
      Assert.assertEquals("false", properties.get("oozie.service.HadoopAccessorService.kerberos.enabled"));

      Map<String, String> updates = new HashMap<>();
      updates.put("oozie.authentication.type", "kerberos");
      updates.put("oozie.service.HadoopAccessorService.kerberos.enabled", "true");

      configHelper.updateConfigType(cluster, managementController, "oozie-site", updates, null, "admin", "Test " +
          "note");

      Config updatedConfig = cluster.getDesiredConfigByType("oozie-site");
      // Config tag updated
      Assert.assertFalse("version1".equals(updatedConfig.getTag()));
      // Property added
      properties = updatedConfig.getProperties();
      Assert.assertTrue(properties.containsKey("oozie.authentication.type"));
      Assert.assertEquals("kerberos", properties.get("oozie.authentication.type"));
      // Property updated
      Assert.assertTrue(properties.containsKey("oozie.service.HadoopAccessorService.kerberos.enabled"));
      Assert.assertEquals("true", properties.get("oozie.service.HadoopAccessorService.kerberos.enabled"));
    }

    @Test
    public void testUpdateConfigTypeRemovals() throws Exception {
      Config currentConfig = cluster.getDesiredConfigByType("ams-site");
      Map<String, String> properties = currentConfig.getProperties();
      // Config tag before update
      Assert.assertEquals("version1", currentConfig.getTag());
      // Properties before update
      Assert.assertEquals("embedded", properties.get("timeline.service.operating.mode"));
      Assert.assertEquals("false", properties.get("timeline.service.fifo.enabled"));

      List<String> removals = new ArrayList<>();
      removals.add("timeline.service.operating.mode");

      configHelper.updateConfigType(cluster, managementController, "ams-site", null, removals, "admin", "Test note");

      Config updatedConfig = cluster.getDesiredConfigByType("ams-site");
      // Config tag updated
      Assert.assertFalse("version1".equals(updatedConfig.getTag()));
      // Property removed
      properties = updatedConfig.getProperties();
      Assert.assertFalse(properties.containsKey("timeline.service.operating.mode"));
      // Property unchanged
      Assert.assertTrue(properties.containsKey("timeline.service.fifo.enabled"));
      Assert.assertEquals("false", properties.get("timeline.service.fifo.enabled"));
    }

    @Test
    public void testCalculateIsStaleConfigs() throws Exception {

      Map<String, HostConfig> schReturn = new HashMap<>();
      HostConfig hc = new HostConfig();
      // Put a different version to check for change
      hc.setDefaultVersionTag("version2");
      schReturn.put("flume-conf", hc);

      // set up mocks
      ServiceComponentHost sch = createNiceMock(ServiceComponentHost.class);
      // set up expectations
      expect(sch.getActualConfigs()).andReturn(schReturn).times(6);
      expect(sch.getHostName()).andReturn("h1").anyTimes();
      expect(sch.getClusterId()).andReturn(cluster.getClusterId()).anyTimes();
      expect(sch.getServiceName()).andReturn("FLUME").anyTimes();
      expect(sch.getServiceComponentName()).andReturn("FLUME_HANDLER").anyTimes();
      replay(sch);
      // Cluster level config changes
      Assert.assertTrue(configHelper.isStaleConfigs(sch, null));

      HostConfig hc2 = new HostConfig();
      hc2.setDefaultVersionTag("version1");
      schReturn.put("flume-conf", hc2);
      // invalidate cache to test new sch
      // Cluster level same configs
      Assert.assertFalse(configHelper.isStaleConfigs(sch, null));

      // Cluster level same configs but group specific configs for host have been updated
      List<String> hosts = new ArrayList<>();
      hosts.add("h1");
      List<Config> configs = new ArrayList<>();

      Config configImpl = configFactory.createNew(cluster, "flume-conf", "FLUME1",
          new HashMap<String,String>(), null);

      configs.add(configImpl);
      addConfigGroup("configGroup1", "FLUME", hosts, configs);

      // config group added for host - expect staleness
      Assert.assertTrue(configHelper.isStaleConfigs(sch, null));

      HostConfig hc3 = new HostConfig();
      hc3.setDefaultVersionTag("version1");
      hc3.getConfigGroupOverrides().put(1l, "FLUME1");
      schReturn.put("flume-conf", hc3);

      // version1 and FLUME1 - stale=false
      Assert.assertFalse(configHelper.isStaleConfigs(sch, null));

      HostConfig hc4 = new HostConfig();
      hc4.setDefaultVersionTag("version1");
      hc4.getConfigGroupOverrides().put(1l, "FLUME2");
      schReturn.put("flume-conf", hc4);

      // version1 and FLUME2 - stale=true
      Assert.assertTrue(configHelper.isStaleConfigs(sch, null));

      HostConfig hc5 = new HostConfig();
      hc5.setDefaultVersionTag("version3");
      hc5.getConfigGroupOverrides().put(1l, "FLUME1");
      schReturn.put("flume-conf", hc5);

      // version3 and FLUME1 - stale=true
      Assert.assertTrue(configHelper.isStaleConfigs(sch, null));

      verify(sch);
    }
  }

  public static class RunWithCustomModule {
    private Injector injector;

    @Before
    public void setup() throws Exception {
      injector = Guice.createInjector(new AbstractModule() {

        @Override
        protected void configure() {
          final AmbariMetaInfo mockMetaInfo = createNiceMock(AmbariMetaInfo.class);
          final ClusterController clusterController = createStrictMock(ClusterController.class);

          bind(UpgradeContextFactory.class).toInstance(createNiceMock(UpgradeContextFactory.class));

          bind(EntityManager.class).toInstance(createNiceMock(EntityManager.class));
          bind(DBAccessor.class).toInstance(createNiceMock(DBAccessor.class));
          bind(ClusterFactory.class).toInstance(createNiceMock(ClusterFactory.class));
          bind(HostFactory.class).toInstance(createNiceMock(HostFactory.class));
          bind(SecurityHelper.class).toInstance(createNiceMock(SecurityHelper.class));
          bind(OsFamily.class).toInstance(createNiceMock(OsFamily.class));
          bind(AmbariCustomCommandExecutionHelper.class).toInstance(createNiceMock(AmbariCustomCommandExecutionHelper.class));
          bind(AmbariManagementController.class).toInstance(createNiceMock(AmbariManagementController.class));
          bind(AmbariMetaInfo.class).toInstance(mockMetaInfo);
          bind(RequestFactory.class).toInstance(createNiceMock(RequestFactory.class));
          bind(Clusters.class).toInstance(createNiceMock(Clusters.class));
          bind(ClusterController.class).toInstance(clusterController);
          bind(StackManagerFactory.class).toInstance(createNiceMock(StackManagerFactory.class));
          bind(HostRoleCommandDAO.class).toInstance(createNiceMock(HostRoleCommandDAO.class));
        }
      });

      // Set the authenticated user
      // TODO: remove this or replace the authenticated user to test authorization rules
      SecurityContextHolder.getContext().setAuthentication(TestAuthenticationFactory.createAdministrator("admin"));
    }

    @After
    public void teardown() {
      // Clear the authenticated user
      SecurityContextHolder.getContext().setAuthentication(null);
    }

    @Test
    public void testGetServicePropertiesSimpleInvocation() throws Exception {
      Cluster mockCluster = createStrictMock(Cluster.class);
      StackId mockStackVersion = createStrictMock(StackId.class);
      AmbariMetaInfo mockAmbariMetaInfo = injector.getInstance(AmbariMetaInfo.class);
      ServiceInfo mockServiceInfo = createStrictMock(ServiceInfo.class);

      PropertyInfo mockPropertyInfo1 = createStrictMock(PropertyInfo.class);
      PropertyInfo mockPropertyInfo2 = createStrictMock(PropertyInfo.class);

      List<PropertyInfo> serviceProperties = Arrays.asList(mockPropertyInfo1, mockPropertyInfo2);

      expect(mockCluster.getCurrentStackVersion()).andReturn(mockStackVersion).once();

      expect(mockStackVersion.getStackName()).andReturn("HDP").once();
      expect(mockStackVersion.getStackVersion()).andReturn("2.2").once();

      expect(mockAmbariMetaInfo.getService("HDP", "2.2", "SERVICE")).andReturn(mockServiceInfo).once();

      expect(mockServiceInfo.getProperties()).andReturn(serviceProperties).once();

      replay(mockAmbariMetaInfo, mockCluster, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);

      mockAmbariMetaInfo.init();

      Set<PropertyInfo> result = injector.getInstance(ConfigHelper.class)
          .getServiceProperties(mockCluster, "SERVICE");

      Assert.assertNotNull(result);
      Assert.assertEquals(2, result.size());

      verify(mockAmbariMetaInfo, mockCluster, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);
    }

    @Test
    public void testGetServicePropertiesDoNoRemoveExcluded() throws Exception {
      StackId mockStackVersion = createStrictMock(StackId.class);
      AmbariMetaInfo mockAmbariMetaInfo = injector.getInstance(AmbariMetaInfo.class);
      ServiceInfo mockServiceInfo = createStrictMock(ServiceInfo.class);

      PropertyInfo mockPropertyInfo1 = createStrictMock(PropertyInfo.class);
      PropertyInfo mockPropertyInfo2 = createStrictMock(PropertyInfo.class);

      List<PropertyInfo> serviceProperties = Arrays.asList(mockPropertyInfo1, mockPropertyInfo2);

      expect(mockStackVersion.getStackName()).andReturn("HDP").once();
      expect(mockStackVersion.getStackVersion()).andReturn("2.2").once();

      expect(mockAmbariMetaInfo.getService("HDP", "2.2", "SERVICE")).andReturn(mockServiceInfo).once();

      expect(mockServiceInfo.getProperties()).andReturn(serviceProperties).once();

      replay(mockAmbariMetaInfo, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);

      mockAmbariMetaInfo.init();

      Set<PropertyInfo> result = injector.getInstance(ConfigHelper.class)
          .getServiceProperties(mockStackVersion, "SERVICE", false);

      Assert.assertNotNull(result);
      Assert.assertEquals(2, result.size());

      verify(mockAmbariMetaInfo, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);
    }

    @Test
    public void testGetServicePropertiesRemoveExcluded() throws Exception {
      StackId mockStackVersion = createStrictMock(StackId.class);
      AmbariMetaInfo mockAmbariMetaInfo = injector.getInstance(AmbariMetaInfo.class);
      ServiceInfo mockServiceInfo = createStrictMock(ServiceInfo.class);

      PropertyInfo mockPropertyInfo1 = createStrictMock(PropertyInfo.class);
      PropertyInfo mockPropertyInfo2 = createStrictMock(PropertyInfo.class);

      List<PropertyInfo> serviceProperties = Arrays.asList(mockPropertyInfo1, mockPropertyInfo2);

      expect(mockStackVersion.getStackName()).andReturn("HDP").once();
      expect(mockStackVersion.getStackVersion()).andReturn("2.2").once();

      expect(mockAmbariMetaInfo.getService("HDP", "2.2", "SERVICE")).andReturn(mockServiceInfo).once();

      expect(mockServiceInfo.getProperties()).andReturn(serviceProperties).once();
      expect(mockServiceInfo.getExcludedConfigTypes()).andReturn(Collections.singleton("excluded-type")).once();

      expect(mockPropertyInfo1.getFilename()).andReturn("included-type.xml").times(2);

      expect(mockPropertyInfo2.getFilename()).andReturn("excluded-type.xml").once();

      replay(mockAmbariMetaInfo, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);

      mockAmbariMetaInfo.init();

      Set<PropertyInfo> result = injector.getInstance(ConfigHelper.class)
          .getServiceProperties(mockStackVersion, "SERVICE", true);

      Assert.assertNotNull(result);
      Assert.assertEquals(1, result.size());
      Assert.assertEquals("included-type.xml", result.iterator().next().getFilename());

      verify(mockAmbariMetaInfo, mockStackVersion, mockServiceInfo, mockPropertyInfo1, mockPropertyInfo2);
    }
  }
}

redpajama_set_name: RedPajamaGithub


text:
"""deCONZ binary sensor platform tests."""
from unittest.mock import Mock, patch

from homeassistant import config_entries
from homeassistant.components import deconz
from homeassistant.helpers.dispatcher import async_dispatcher_send
from homeassistant.setup import async_setup_component

import homeassistant.components.binary_sensor as binary_sensor

from tests.common import mock_coro

SENSOR = {
    "1": {
        "id": "Sensor 1 id",
        "name": "Sensor 1 name",
        "type": "ZHAPresence",
        "state": {"presence": False},
        "config": {},
        "uniqueid": "00:00:00:00:00:00:00:00-00"
    },
    "2": {
        "id": "Sensor 2 id",
        "name": "Sensor 2 name",
        "type": "ZHATemperature",
        "state": {"temperature": False},
        "config": {}
    }
}

ENTRY_CONFIG = {
    deconz.const.CONF_ALLOW_CLIP_SENSOR: True,
    deconz.const.CONF_ALLOW_DECONZ_GROUPS: True,
    deconz.config_flow.CONF_API_KEY: "ABCDEF",
    deconz.config_flow.CONF_BRIDGEID: "0123456789",
    deconz.config_flow.CONF_HOST: "1.2.3.4",
    deconz.config_flow.CONF_PORT: 80
}

async def setup_gateway(hass, data, allow_clip_sensor=True):
    """Load the deCONZ binary sensor platform."""
    from pydeconz import DeconzSession
    loop = Mock()
    session = Mock()

    ENTRY_CONFIG[deconz.const.CONF_ALLOW_CLIP_SENSOR] = allow_clip_sensor

    config_entry = config_entries.ConfigEntry(
        1, deconz.DOMAIN, 'Mock Title', ENTRY_CONFIG, 'test',
        config_entries.CONN_CLASS_LOCAL_PUSH)
    gateway = deconz.DeconzGateway(hass, config_entry)
    gateway.api = DeconzSession(loop, session, **config_entry.data)
    gateway.api.config = Mock()
    hass.data[deconz.DOMAIN] = gateway

    with patch('pydeconz.DeconzSession.async_get_state',
               return_value=mock_coro(data)):
        await gateway.api.async_load_parameters()

    await hass.config_entries.async_forward_entry_setup(
        config_entry, 'binary_sensor')
    # To flush out the service call to update the group
    await hass.async_block_till_done()

async def test_platform_manually_configured(hass):
    """Test that we do not discover anything or try to set up a gateway."""
    assert await async_setup_component(hass, binary_sensor.DOMAIN, {
        'binary_sensor': {
            'platform': deconz.DOMAIN
        }
    }) is True
    assert deconz.DOMAIN not in hass.data

async def test_no_binary_sensors(hass):
    """Test that no sensors in deconz results in no sensor entities."""
    data = {}
    await setup_gateway(hass, data)
    assert len(hass.data[deconz.DOMAIN].deconz_ids) == 0
    assert len(hass.states.async_all()) == 0

async def test_binary_sensors(hass):
    """Test successful creation of binary sensor entities."""
    data = {"sensors": SENSOR}
    await setup_gateway(hass, data)
    assert "binary_sensor.sensor_1_name" in \
        hass.data[deconz.DOMAIN].deconz_ids
    assert "binary_sensor.sensor_2_name" not in \
        hass.data[deconz.DOMAIN].deconz_ids
    assert len(hass.states.async_all()) == 1

    hass.data[deconz.DOMAIN].api.sensors['1'].async_update(
        {'state': {'on': False}})

async def test_add_new_sensor(hass):
    """Test successful creation of sensor entities."""
    data = {}
    await setup_gateway(hass, data)
    sensor = Mock()
    sensor.name = 'name'
    sensor.type = 'ZHAPresence'
    sensor.register_async_callback = Mock()
    async_dispatcher_send(hass, 'deconz_new_sensor', [sensor])
    await hass.async_block_till_done()
    assert "binary_sensor.name" in hass.data[deconz.DOMAIN].deconz_ids

async def test_do_not_allow_clip_sensor(hass):
    """Test that clip sensors can be ignored."""
    data = {}
    await setup_gateway(hass, data, allow_clip_sensor=False)
    sensor = Mock()
    sensor.name = 'name'
    sensor.type = 'CLIPPresence'
    sensor.register_async_callback = Mock()
    async_dispatcher_send(hass, 'deconz_new_sensor', [sensor])
    await hass.async_block_till_done()
    assert len(hass.data[deconz.DOMAIN].deconz_ids) == 0

async def test_unload_switch(hass):
    """Test that it works to unload switch entities."""
    data = {"sensors": SENSOR}
    await setup_gateway(hass, data)

    await hass.data[deconz.DOMAIN].async_reset()

    assert len(hass.states.async_all()) == 0


text:
<?xml version="1.0" encoding="utf-8"?><!--
  ~ Copyright 2015 marlonlom
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/container"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:clipChildren="false"
    android:fitsSystemWindows="true"
    tools:context=".MainActivity">

    <fragment
        android:id="@+id/fragment_feed_list"
        android:name="co.malm.mglam_reads.mobile.fragments.FeedListFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        tools:layout="@layout/fragment_feed_list" />

    <include
        layout="@layout/toolbar_default"
        android:layout_width="match_parent"
        android:layout_height="wrap_content" />

</FrameLayout>


text:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>JCatascopia</groupId>
    <artifactId>JCatascopia-Probe-Repo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
  </parent>
  <artifactId>CassandraProbe</artifactId>
  <dependencies>
  <dependency>
  	<groupId>JCatascopia</groupId>
  	<artifactId>ProbePack</artifactId>
  	<version>0.0.1-SNAPSHOT</version>
  </dependency>
  </dependencies>
    <build>
  	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-shade-plugin</artifactId>
			<version>2.1</version>
			<executions>
				<execution>
					<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
					<configuration>
						<transformers>
							<transformer
									implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
									<mainClass>CassandraProbe</mainClass>
							</transformer>
						</transformers>
					</configuration>
				</execution>
			</executions>
		</plugin>
	</plugins>
  </build>
</project>


text:
module P

  class InterpStringExpr < Expr

    def evaluate( environment )
      s = list.inject( '' ) { |m,n| m + n.evaluate( environment ).to_s }
      P.string( s )
    end

  end

end


text:
package com.gilt.gfc.collection

import org.scalatest.{Matchers, FunSuite}

class CircularBufferTest extends FunSuite with Matchers {

  test("Basics") {
    val buffer = new CircularBuffer[Int](5)

    buffer.toIterator.toList should equal(List())
    buffer.size should equal(0)
    an [AssertionError] should be thrownBy buffer.oldest
    an [AssertionError] should be thrownBy  buffer.newest

    buffer.add(1)
    buffer.toIterator.toList should equal(List(1))
    buffer.size should equal(1)
    buffer.oldest should  equal(1)
    buffer.newest should equal(1)

    buffer.add(2)
    buffer.toIterator.toList should equal(List(1,2))
    buffer.size should equal(2)
    buffer.oldest should  equal(1)
    buffer.newest should equal(2)

    buffer.add(3)
    buffer.toIterator.toList should equal(List(1,2,3))
    buffer.size should equal(3)
    buffer.oldest should  equal(1)
    buffer.newest should equal(3)

    buffer.add(4)
    buffer.toIterator.toList should equal(List(1,2,3,4))
    buffer.size should equal(4)
    buffer.oldest should  equal(1)
    buffer.newest should equal(4)

    buffer.add(5)
    buffer.toIterator.toList should equal(List(1,2,3,4,5))
    buffer.size should equal(5)
    buffer.oldest should  equal(1)
    buffer.newest should equal(5)

    buffer.add(6)
    buffer.toIterator.toList should equal(List(2,3,4,5,6))
    buffer.size should equal(5)
    buffer.oldest should  equal(2)
    buffer.newest should equal(6)
  }
}


text:
package org.instantlogic.designer.placetemplate;

public abstract class AbstractOpenEditorPlaceTemplate extends org.instantlogic.interaction.flow.PlaceTemplate {

	public static final OpenEditorPlaceTemplate INSTANCE = new OpenEditorPlaceTemplate();

	@Override
	public String getId() {
		return "PlaceTemplateDesign_OpenEditor";
	}

	private static org.instantlogic.interaction.page.FragmentTemplate CONTENT =
		null;

	private static final org.instantlogic.fabric.model.Entity[] PARAMETERS = new org.instantlogic.fabric.model.Entity[]{
		org.instantlogic.designer.entity.ElementDesignEntity.INSTANCE,
	};

	@Override
	public org.instantlogic.fabric.model.Entity[] getParameters() {
		return PARAMETERS;
	}

	@Override
	public org.instantlogic.interaction.page.FragmentTemplate getRootContainer() {
		return CONTENT;
	};

    @Override
    public String getName() {
        return "OpenEditor";
    }

    @Override
    public String getTechnicalName() {
        return "OpenEditor";
    }
}


text:
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

namespace DynamicsConnector
{
    public class UppercaseGuidConverter : JsonConverter
    {
        public override bool CanConvert(Type objectType)
        {
            return (objectType == typeof(Guid) || objectType == typeof(Guid?));
        }

        public override bool CanRead
        {
            get
            {
                return false;
            }
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            if (value == null)
            {
                writer.WriteNull();
            }

            writer.WriteValue(value.ToString().ToUpperInvariant());
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            throw new NotImplementedException("Should never be called because CanRead returns false.");
        }
    }
}


text:
const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

const PROJECT_ID = process.env.PROJECT_ID
const DATASET_ID = process.env.DATASET_ID
const MESSAGES_TABLE = process.env.MESSAGES_TABLE
const TELEMETRY_TABLE = process.env.TELEMETRY_TABLE

const STATE = 1
const EVENT_POINTSET = 2
const EVENT_SYSTEM = 3
const EVENT_DISCOVERY = 4
const EVENT_OTHER = 9

exports.processMessage = async (event, context) => {
  // Only process messages from device and ignore all message fragments
  if (event.attributes.subType == 'state' && event.attributes.subFolder == 'update') {
    var messageType = STATE
  } else if (!event.attributes.hasOwnProperty('subType')) {
    if (event.attributes.subFolder == 'pointset'){
      var messageType = EVENT_POINTSET;
    } else if (event.attributes.subFolder == 'pointset'){
      var messageType = EVENT_SYSTEM;
    } else {
      var messageType = EVENT_OTHER
    }
  } else {
    return;
  }
  const pubsubMessage = event.data;
  const objStr = Buffer.from(pubsubMessage, 'base64').toString('utf8');
  const msgObj = JSON.parse(objStr);

  // This is a Pub/Sub message ID -  messages copied in the stack
  // will have different message IDs
  const messageId = context.eventId;

  const publishTimestamp = BigQuery.timestamp(context.timestamp);
  const deviceId = event.attributes.deviceId;
  const gatewayId = ("gatewayId" in event.attributes) && event.attributes.gatewayId || null;
  const deviceRegistryId = event.attributes.deviceRegistryId;

  const payloadSize = Buffer.byteLength(objStr, 'utf8');

  var promises = [];

  // add message to table
  var messageRow = {
    publish_timestamp: publishTimestamp,
    device_num_id: parseInt(event.attributes.deviceNumId),
    device_id: deviceId,
    gateway_id: gatewayId,
    registry_id: deviceRegistryId,
    message_id: messageId,
    payload_size_bytes: payloadSize,
    message_type: messageType
  };

  promises.push(bigquery.dataset(DATASET_ID).table(MESSAGES_TABLE).insert([messageRow]));

  // Insert Telemetry
  if ('points' in msgObj){
    var rows = []
    const payloadTimestamp = BigQuery.timestamp(msgObj.timestamp);
    Object.keys(msgObj.points).forEach(function(key) {
      let row = {
        device_id: deviceId,
        device_num_id: parseInt(event.attributes.deviceNumId),
        message_id: messageId,
        gateway_id: gatewayId,
        registry_id: deviceRegistryId,
        publish_timestamp: publishTimestamp,
        timestamp: payloadTimestamp,
        point_name: key,
        present_value: (isNaN(parseFloat(msgObj.points[key].present_value)) ? null : parseFloat(msgObj.points[key].present_value) ),
        present_value_string: String(msgObj.points[key].present_value)
      };
        rows.push(row)
    });

     if (rows.length > 0){
      promises.push(bigquery.dataset(DATASET_ID).table(TELEMETRY_TABLE).insert(rows))
     }

  }

  return await Promise.all(promises);
};


text:
module Asposetasksjava
  module RenderTaskUsageView
    def initialize()
        data_dir = File.dirname(File.dirname(File.dirname(File.dirname(__FILE__)))) + '/data/'

        # Instantiate project object
        project = Rjb::import('com.aspose.tasks.Project').new(data_dir + 'test_tasks.mpp')

        # Define the SaveOptions with required TimeScale settings as Days
		options = Rjb::import('com.aspose.tasks.PdfSaveOptions').new
		options.setTimescale(Rjb::import('com.aspose.tasks.Timescale').Days)
		# Set the Presentation format to ResourceUsage
		options.setPresentationFormat(Rjb::import('com.aspose.tasks.PresentationFormat').TaskUsage)
		# Save the Project
		project.save(data_dir + "task_days.pdf", options)

		# Set the Tiemscale settings to ThirdsOfMonths
		options.setTimescale(Rjb::import('com.aspose.tasks.Timescale').ThirdsOfMonths)
		# Save the Project
		project.save(data_dir + "task_thirdsOfMonths.pdf", options)

		# Set the Timescale settings to Months
		options.setTimescale(Rjb::import('com.aspose.tasks.Timescale').Months)
		# Save the project
		project.save(data_dir + "task_months.pdf", options)

		puts "Created task usage view files."
    end
  end
end


text:
package com.azakordonets.web.entities;

public class User {
    private final String email;
    private String newPassword;
    private long resetPasswordTokenTs;

    public User(String email) {
        this.email = email;
        this.newPassword = "";
        this.resetPasswordTokenTs = System.currentTimeMillis();
    }

    public String getEmail() {
        return email;
    }

    public String getNewPassword() {
        return newPassword;
    }

    public void setNewPassword(String newPassword) {
        this.newPassword = newPassword;
    }

    public long getResetPasswordTokenTs() {
        return resetPasswordTokenTs;
    }

    public void setResetPasswordTokenTs(long resetPasswordTokenTs) {
        this.resetPasswordTokenTs = resetPasswordTokenTs;
    }
}


text:
using System;
using System.Collections.Generic;
using System.IO;
using Swartz.Caching;

namespace Swartz.FileSystems.AppData
{
    public interface IAppDataFolder : IVolatileProvider
    {
        IEnumerable<string> ListFiles(string path);
        IEnumerable<string> ListDirectories(string path);

        string Combine(params string[] paths);

        bool FileExists(string path);

        ///
        ///     Creates or overwrites the file in the specified path with the specified content.
        ///
        /// <param name="path">The path and name of the file to create.</param>
        /// <param name="content">The content to write in the created file.</param>
        /// <remarks>If the folder doesn't exist, it will be created.</remarks>
        void CreateFile(string path, string content);

        ///
        ///     Creates or overwrites the file in the specified path.
        ///
        /// <param name="path">The path and name of the file to create.</param>
        /// <returns>
        ///     A <see cref="Stream" /> that provides read/write access to the file specified in path.
        /// </returns>
        /// <remarks>If the folder doesn't exist, it will be created.</remarks>
        Stream CreateFile(string path);

        ///
        ///     Opens a text file, reads all lines of the file, and then closes the file.
        ///
        /// <param name="path">The path and name of the file to read.</param>
        /// <returns>A string containing all lines of the file, or null if the file doesn't exist.</returns>
        string ReadFile(string path);

        ///
        ///     Open an existing file for reading.
        ///
        /// <param name="path">The path and name of the file to create.</param>
        /// <returns>
        ///     A <see cref="Stream" /> that provides read access to the file specified in path.
        /// </returns>
        Stream OpenFile(string path);

        void StoreFile(string sourceFileName, string destinationPath);
        void DeleteFile(string path);

        DateTime GetFileLastWriteTimeUtc(string path);

        void CreateDirectory(string path);
        bool DirectoryExists(string path);

        IVolatileToken WhenPathChanges(string path);

        string MapPath(string path);
        string GetVirtualPath(string path);
    }
}


text:
#ifndef INCLUDED_IMF_RGBA_YCA_H
#define INCLUDED_IMF_RGBA_YCA_H

//////////////////////////////////////////////////////////////////////////////
//
// Copyright (c) 2004, Industrial Light & Magic, a division of Lucasfilm
// Entertainment Company Ltd.  Portions contributed and copyright held by
// others as indicated.  All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are
// met:
//
//     * Redistributions of source code must retain the above
//       copyright notice, this list of conditions and the following
//       disclaimer.
//
//     * Redistributions in binary form must reproduce the above
//       copyright notice, this list of conditions and the following
//       disclaimer in the documentation and/or other materials provided with
//       the distribution.
//
//     * Neither the name of Industrial Light & Magic nor the names of
//       any other contributors to this software may be used to endorse or
//       promote products derived from this software without specific prior
//       written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
// IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
// THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
// PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
// CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
// EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
// PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
// LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
//////////////////////////////////////////////////////////////////////////////

//-----------------------------------------------------------------------------
//
//	Conversion between RGBA (red, green, blue alpha)
//	and YCA (luminance, subsampled chroma, alpha) data:
//
//	Luminance, Y, is computed as a weighted sum of R, G, and B:
//
//		Y = yw.x * R + yw.y * G + yw.z * B
//
//	Function computeYw() computes a set of RGB-to-Y weights, yw,
//	from a set of primary and white point chromaticities.
//
//	Chroma, C, consists of two components, RY and BY:
//
//		RY = (R - Y) / Y
//		BY = (B - Y) / Y
//
//	For efficiency, the x and y subsampling rates for chroma are
//	hardwired to 2, and the chroma subsampling and reconstruction
//	filters are fixed 27-pixel wide windowed sinc functions.
//
//	Starting with an image that has RGBA data for all pixels,
//
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//		...
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//		RGBA RGBA RGBA RGBA ... RGBA RGBA
//
//	function RGBAtoYCA() converts the pixels to YCA format:
//
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//		...
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//		YCA  YCA  YCA  YCA  ... YCA  YCA
//
//	Next, decimateChomaHoriz() eliminates the chroma values from
//	the odd-numbered pixels in every scan line:
//
//		YCA  YA   YCA  YA   ... YCA  YA
//		YCA  YA   YCA  YA   ... YCA  YA
//		YCA  YA   YCA  YA   ... YCA  YA
//		YCA  YA   YCA  YA   ... YCA  YA
//		...
//		YCA  YA   YCA  YA   ... YCA  YA
//		YCA  YA   YCA  YA   ... YCA  YA
//
//	decimateChromaVert() eliminates all chroma values from the
//	odd-numbered scan lines:
//
//		YCA  YA   YCA  YA   ... YCA  YA
//		YA   YA   YA   YA   ... YA   YA
//		YCA  YA   YCA  YA   ... YCA  YA
//		YA   YA   YA   YA   ... YA   YA
//		...
//		YCA  YA   YCA  YA   ... YCA  YA
//		YA   YA   YA   YA   ... YA   YA
//
//	Finally, roundYCA() reduces the precision of the luminance
//	and chroma values so that the pixel data shrink more when
//	they are saved in a compressed file.
//
//	The output of roundYCA() can be converted back to a set
//	of RGBA pixel data that is visually very similar to the
//	original RGBA image, by calling reconstructChromaHoriz(),
//	reconstructChromaVert(), YCAtoRGBA(), and finally
//	fixSaturation().
//
//-----------------------------------------------------------------------------

#include <ImfRgba.h>
#include BOSS_OPENCV_V_ImfChromaticities_h //original-code:<ImfChromaticities.h>

namespace Imf {
namespace RgbaYca {

//
// Width of the chroma subsampling and reconstruction filters
//

static const int N = 27;
static const int N2 = N / 2;

//
// Convert a set of primary chromaticities into a set of weighting
// factors for computing a pixels's luminance, Y, from R, G and B
//

Imath::V3f computeYw (const Chromaticities &cr);

//
// Convert an array of n RGBA pixels, rgbaIn, to YCA (luminance/chroma/alpha):
//
//	ycaOut[i].g = Y (rgbaIn[i]);
//	ycaOut[i].r = RY (rgbaIn[i]);
//	ycaOut[i].b = BY (rgbaIn[i]);
//	ycaOut[i].a = aIsValid? rgbaIn[i].a: 1
//
// yw is a set of RGB-to-Y weighting factors, as computed by computeYw().
//

void RGBAtoYCA (const Imath::V3f &yw,
        int n,
            bool aIsValid,
        const Rgba rgbaIn[/*n*/],
        Rgba ycaOut[/*n*/]);

//
// Perform horizontal low-pass filtering and subsampling of
// the chroma channels of an array of n pixels.  In order
// to avoid indexing off the ends of the input array during
// low-pass filtering, ycaIn must have N2 extra pixels at
// both ends.  Before calling decimateChromaHoriz(), the extra
// pixels should be filled with copies of the first and last
// "real" input pixel.
//

void decimateChromaHoriz (int n,
              const Rgba ycaIn[/*n+N-1*/],
              Rgba ycaOut[/*n*/]);

//
// Perform vertical chroma channel low-pass filtering and subsampling.
// N scan lines of input pixels are combined into a single scan line
// of output pixels.
//

void decimateChromaVert (int n,
             const Rgba * const ycaIn[N],
             Rgba ycaOut[/*n*/]);

//
// Round the luminance and chroma channels of an array of YCA
// pixels that has already been filtered and subsampled.
// The signifcands of the pixels' luminance and chroma values
// are rounded to roundY and roundC bits respectively.
//

void roundYCA (int n,
           unsigned int roundY,
           unsigned int roundC,
           const Rgba ycaIn[/*n*/],
           Rgba ycaOut[/*n*/]);

//
// For a scan line that has valid chroma data only for every other pixel,
// reconstruct the missing chroma values.
//

void reconstructChromaHoriz (int n,
                 const Rgba ycaIn[/*n+N-1*/],
                 Rgba ycaOut[/*n*/]);

//
// For a scan line that has only luminance and no valid chroma data,
// reconstruct chroma from the surronding N scan lines.
//

void reconstructChromaVert (int n,
                const Rgba * const ycaIn[N],
                Rgba ycaOut[/*n*/]);

//
// Convert an array of n YCA (luminance/chroma/alpha) pixels to RGBA.
// This function is the inverse of RGBAtoYCA().
// yw is a set of RGB-to-Y weighting factors, as computed by computeYw().
//

void YCAtoRGBA (const Imath::V3f &yw,
        int n,
        const Rgba ycaIn[/*n*/],
        Rgba rgbaOut[/*n*/]);

//
// Eliminate super-saturated pixels:
//
// Converting an image from RGBA to YCA, low-pass filtering chroma,
// and converting the result back to RGBA can produce pixels with
// super-saturated colors, where one or two of the RGB components
// become zero or negative.  (The low-pass and reconstruction filters
// introduce some amount of ringing into the chroma components.
// This can lead to negative RGB values near high-contrast edges.)
//
// The fixSaturation() function finds super-saturated pixels and
// corrects them by desaturating their colors while maintaining
// their luminance.  fixSaturation() takes three adjacent input
// scan lines, rgbaIn[0], rgbaIn[1], rgbaIn[2], adjusts the
// saturation of rgbaIn[1], and stores the result in rgbaOut.
//

void fixSaturation (const Imath::V3f &yw,
            int n,
            const Rgba * const rgbaIn[3],
            Rgba rgbaOut[/*n*/]);

} // namespace RgbaYca
} // namespace Imf

#endif


text:
package com.example;

public class F {
    public String greet() {
        return "Good morning";
    }
}


text:
require File.expand_path('../../test_helper', __FILE__)

class ContactsMailerTest < ActiveSupport::TestCase
  fixtures :projects,
           :users,
           :roles,
           :members,
           :member_roles,
           :issues,
           :issue_statuses,
           :versions,
           :trackers,
           :projects_trackers,
           :issue_categories,
           :enabled_modules,
           :enumerations,
           :attachments,
           :workflows,
           :custom_fields,
           :custom_values,
           :custom_fields_projects,
           :custom_fields_trackers,
           :time_entries,
           :journals,
           :journal_details,
           :queries

    ActiveRecord::Fixtures.create_fixtures(File.dirname(__FILE__) + '/../fixtures/',
                            [:contacts,
                             :contacts_projects,
                             :contacts_issues,
                             :deals,
                             :notes,
                             :roles,
                             :enabled_modules,
                             :tags,
                             :taggings,
                             :contacts_queries])

  FIXTURES_PATH = File.dirname(__FILE__) + '/../fixtures/contacts_mailer'

  def setup
    RedmineContacts::TestCase.prepare

    ActionMailer::Base.deliveries.clear
    Setting.notified_events = Redmine::Notifiable.all.collect(&:name)
  end

  test "Should add contact note from to" do
    # This email contains: 'Project: onlinestore'
    note = submit_email('new_note.eml').first
    assert_instance_of ContactNote, note
    assert !note.new_record?
    note.reload
    assert_equal Contact, note.source.class
    assert_equal "New note from email", note.subject
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Contact.find(1).id, note.source_id
  end

  test "Should add contact note from ID in to" do
    # This email contains: 'Project: onlinestore'
    note = submit_email('new_note_by_id.eml').first
    assert_instance_of ContactNote, note
    assert !note.new_record?
    note.reload
    assert_equal Contact, note.source.class
    assert_equal "New note from email", note.subject
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Contact.find(1).id, note.source_id
  end

  test "Should add contact note from ID in cc" do
    # This email contains: 'Project: onlinestore'
    note = submit_email('new_note_with_cc.eml').first
    assert_instance_of ContactNote, note
    assert !note.new_record?
    note.reload
    assert_equal Contact, note.source.class
    assert_equal "New note from email by id in cc", note.subject
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Contact.find(1).id, note.source_id
  end

  test "Should add deal note from ID in to" do
    # This email contains: 'Project: onlinestore'
    note = submit_email('new_deal_note_by_id.eml').first
    assert_instance_of DealNote, note
    assert !note.new_record?
    note.reload
    assert_equal Deal, note.source.class
    assert_equal "New note from email", note.subject
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Deal.find(1).id, note.source_id
  end

  test "Should add contact note from forwarded" do
    note = submit_email('fwd_new_note_plain.eml').first
    assert_instance_of ContactNote, note
    assert !note.new_record?
    note.reload
    assert_equal Contact, note.source.class
    assert_equal "New note from forwarded email", note.subject
    assert_match 'From: "Marat Aminov" marat@mail.ru', note.content
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Contact.find(2).id, note.source_id
  end

  test "Should add contact note from forwarded html" do
    note = submit_email('fwd_new_note_html.eml').first
    assert_instance_of ContactNote, note
    assert !note.new_record?
    note.reload
    assert_equal Contact, note.source.class
    assert_equal "New note from forwarded html email", note.subject
    assert_match "From: Marat Aminov <marat@mail.com>", note.content
    assert_equal User.find_by_login('admin'), note.author
    assert_equal Contact.find(2).id, note.source_id
  end

  test "Should not add contact note from deny user to" do
    assert !submit_email('new_deny_note.eml')
  end

  private

  def submit_email(filename, options={})
    raw = IO.read(File.join(FIXTURES_PATH, filename))
    ContactsMailer.receive(raw, options)
  end

end


text:
<?php

// Email address verification
function isEmail($email) {
	return filter_var($email, FILTER_VALIDATE_EMAIL);
}

if($_POST) {

    // Enter the email where you want to receive the message
    $emailTo = 'auntmaryrva@gmail.com';

    $name = addslashes(trim($_POST['name']));
    $clientEmail = addslashes(trim($_POST['email']));
    $subject = addslashes(trim($_POST['subject']));
    $message = addslashes(trim($_POST['message']));

    $array = array('nameMessage' => '', 'emailMessage' => '', 'subjectMessage' => '', 'messageMessage' => '');

    if($name == '') {
    	$array['nameMessage'] = 'Empty name!';
    }
    if(!isEmail($clientEmail)) {
        $array['emailMessage'] = 'Invalid email!';
    }
    if($subject == '') {
        $array['subjectMessage'] = 'Empty subject!';
    }
    if($message == '') {
        $array['messageMessage'] = 'Empty message!';
    }
    if($name != '' && isEmail($clientEmail) && $subject != '' && $message != '') {
		// Send email
		$message = "Message from: " . $name . "\r\n" . $message;
		$headers = "From: " . $clientEmail . " <" . $clientEmail . ">" . "\r\n" . "Reply-To: " . $clientEmail;
		mail($emailTo, $subject . " (contact form template)", $message, $headers);
    }

    echo json_encode($array);

}

?>


text:
from PyQt4 import QtCore
import urllib
import json

__author__ = 'postrowski'

# -*-coding: utf-8-*-

class FindSongThread(QtCore.QThread):

    def __init__(self, parent):
        super(FindSongThread, self).__init__(parent)

        self.get_album_tracklist = ''

        self.albumList = parent.albumList
        self.songList = parent.songList

        self.album_dict = parent.album.album_dict

    def find_song(self):
        """
            Find album songs.
        """
        self.songList.clear()

        # get checked album title...
        get_alb_name = QtCore.QString(unicode(self.albumList.currentItem().text()))
        # ...and get his tracklist (two or more same album names; need to iterate by indexes)
        row = self.albumList.currentRow()
        self.get_album_tracklist = ''.join([y for x, y in enumerate(self.album_dict.keys()) if row is x])

        # console information
        print(u"Search for tracks from {0}.".format(unicode(get_alb_name)))
        print(self.get_album_tracklist)

        # open json file
        json_url = urllib.urlopen(self.get_album_tracklist)
        data = json.load(json_url)

        # get values from json file
        sng_title_list = []

        for i in range(len(data["data"])):
            sng_title = data["data"][i]["title"]
            sng_title_list.append(sng_title)

        json_url.close()

        # paste songs to list
        [self.songList.addItem(sng_title_list[i]) for i in range(len(sng_title_list))]

        return self.get_album_tracklist

    def run(self):
        self.find_song()
        return


text:
<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: dcm4chee-forward2-service.xml 3310 2007-05-06 23:42:06Z gunterze $ -->
<server>
  <classpath codebase="lib" archives="dcm4chee.jar"/>
  <mbean code="org.dcm4chex.archive.dcm.movescu.ForwardByQueryService"
    name="dcm4chee.archive:service=ForwardByQuery"
    xmbean-dd="xmdesc/dcm4chee-forwardquery-xmbean.xml">
    <depends optional-attribute-name="MoveScuServiceName"
    	>dcm4chee.archive:service=MoveScu</depends>
	<depends optional-attribute-name="SchedulerServiceName"
		>dcm4chee.archive:service=Scheduler</depends>
	<!--  <depends>jboss.jdbc:datasource=pacsDS,service=metadata</depends>
	<depends>jboss.j2ee:jndiName=ejb/MPPSEmulator,service=EJB</depends> -->
  </mbean>
</server>


text:
//=====================================================================
//
//File:      $RCSfile: OpenCanvasEditor.java,v $
//Version:   $Revision: 1.30 $
//Modified:  $Date: 2013/05/10 05:41:50 $
//
//(c) Copyright 2004-2014 by Mentor Graphics Corp. All rights reserved.
//
//=====================================================================
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License.  You may obtain a copy
// of the License at
//
//       http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.   See the
// License for the specific language governing permissions and limitations under
// the License.
//=====================================================================

package com.mentor.nucleus.bp.ui.canvas.test;

import org.eclipse.ui.PlatformUI;

import com.mentor.nucleus.bp.core.InstanceStateMachine_c;
import com.mentor.nucleus.bp.core.ModelClass_c;
import com.mentor.nucleus.bp.core.Package_c;
import com.mentor.nucleus.bp.core.common.ClassQueryInterface_c;
import com.mentor.nucleus.bp.test.common.BaseTest;
import com.mentor.nucleus.bp.test.common.CanvasTestUtils;
import com.mentor.nucleus.bp.ui.graphics.editor.GraphicalEditor;
import com.mentor.nucleus.bp.ui.graphics.editor.ModelEditor;

public class OpenCanvasEditor extends BaseTest {

    private static boolean initialized = false;

	public OpenCanvasEditor(String arg0) {
		super("com.mentor.nucleus.bp.ui.canvas.test", arg0);
	}

	protected void setUp() throws Exception {
		super.setUp();

        if ( !initialized )
        {
            initialized = true;

            // reload model used by this class
            loadProject("odms");
        }

	}

	protected void tearDown() throws Exception {
		super.tearDown();
	}

	public static void validateCanvasEditor(String title, String contents) {
		GraphicalEditor ce = ((ModelEditor) PlatformUI.getWorkbench()
				.getActiveWorkbenchWindow().getActivePage().getActiveEditor())
				.getGraphicalEditor();

		assertNotNull (ce);
		assertEquals( title, ce.getTitle());
	// What is equivalent for Canvas ?
		//assertEquals( contents, ce.getDocumentProvider().getDocument(ce.getEditorInput()).get());

	}

	public void testOpenPackageDiagram()
	{
		final Package_c uut = Package_c.PackageInstance(modelRoot, new ClassQueryInterface_c() {

			@Override
			public boolean evaluate(Object candidate) {
				return ((Package_c) candidate).getName().equals("odms");
			}
		});
		CanvasTestUtils.openDiagramEditor(uut);
		validateCanvasEditor("odms: Package Diagram", uut.getDescrip());
	}

	public void testOpenInstanceStateChartDiagram()
	{
		CanvasTestUtils ctu = new CanvasTestUtils();
		final InstanceStateMachine_c uut = InstanceStateMachine_c.getOneSM_ISMOnR518(ModelClass_c.ModelClassInstance(modelRoot, ctu.new Class_by_name_c("Disk")));
		CanvasTestUtils.openDiagramEditor(uut);
		validateCanvasEditor( "Disk: Instance State Machine", uut.Get_name());

	}

}