text:
<!DOCTYPE html >

          <title>Conductors - ScalaTest 3.0.0 - org.scalatest.concurrent.Conductors</title>
          <meta name="description" content="Conductors - ScalaTest 3.0.0 - org.scalatest.concurrent.Conductors" />
          <meta name="keywords" content="Conductors ScalaTest 3.0.0 org.scalatest.concurrent.Conductors" />
          <meta http-equiv="content-type" content="text/html; charset=UTF-8" />

      <link href="../../../lib/template.css" media="screen" type="text/css" rel="stylesheet" />
      <link href="../../../lib/diagrams.css" media="screen" type="text/css" rel="stylesheet" id="diagrams-css" />

org.scalatest.concurrent

Conductors

Related Doc:
            package concurrent

        trait

        Conductors extends PatienceConfiguration

Trait whose Conductor member facilitates the testing of classes, traits, and libraries designed
to be used by multiple threads concurrently.

A Conductor conducts a multi-threaded scenario by maintaining
a clock of "beats." Beats are numbered starting with 0. You can ask a
Conductor to run threads that interact with the class, trait,
or library (the subject)
you want to test. A thread can call the Conductor's
waitForBeat method, which will cause the thread to block
until that beat has been reached. The Conductor will advance
the beat only when all threads participating in the test are blocked. By
tying the timing of thread activities to specific beats, you can write
tests for concurrent systems that have deterministic interleavings of
threads.

A Conductor object has a three-phase lifecycle. It begins its life
in the setup phase. During this phase, you can start threads by
invoking the thread method on the Conductor.
When conduct is invoked on a Conductor, it enters
the conducting phase. During this phase it conducts the one multi-threaded
scenario it was designed to conduct. After all participating threads have exited, either by
returning normally or throwing an exception, the conduct method
will complete, either by returning normally or throwing an exception. As soon as
the conduct method completes, the Conductor
enters its defunct phase. Once the Conductor has conducted
a multi-threaded scenario, it is defunct and can't be reused. To run the same test again,
you'll need to create a new instance of Conductor.

Here's an example of the use of Conductor to test the ArrayBlockingQueue
class from java.util.concurrent:

import org.scalatest.fixture.FunSuite
import org.scalatest.matchers.Matchers
import java.util.concurrent.ArrayBlockingQueue
import org.scalatest.concurrent.Conductors

class ArrayBlockingQueueSuite extends FunSuite with Matchers with Conductors {

  test("calling put on a full queue blocks the producer thread") {

    val conductor = new Conductor
    import conductor._

    val buf = new ArrayBlockingQueue[Int](1)

    thread("producer") {
      buf put 42
      buf put 17
      beat should be (1)
    }

    thread("consumer") {
      waitForBeat(1)
      buf.take should be (42)
      buf.take should be (17)
    }

    whenFinished {
      buf should be ('empty)
    }
  }
}

When the test shown is run, it will create one thread named producer and another named
consumer. The producer thread will eventually execute the code passed as a by-name
parameter to thread("producer"):

buf put 42
buf put 17
beat should be (1)

Similarly, the consumer thread will eventually execute the code passed as a by-name parameter
to thread("consumer"):

waitForBeat(1)
buf.take should be (42)
buf.take should be (17)

The thread creates the threads and starts them, but they will not immediately
execute the by-name parameter passed to them. They will first block, waiting for the Conductor
to give them a green light to proceed.

The next call in the test is whenFinished. This method will first call conduct on
the Conductor, which will wait until all threads that were created (in this case, producer and consumer) are
at the "starting line", i.e., they have all started and are blocked, waiting on the green light.
The conduct method will then give these threads the green light and they will
all start executing their blocks concurrently.

When the threads are given the green light, the beat is 0. The first thing the producer thread does is put 42 in
into the queue. As the queue is empty at this point, this succeeds. The producer thread next attempts to put a 17
into the queue, but because the queue has size 1, this can't succeed until the consumer thread has read the 42
from the queue. This hasn't happened yet, so producer blocks. Meanwhile, the consumer thread's first act is to
call waitForBeat(1). Because the beat starts out at 0, this call will block the consumer thread.
As a result, once the producer thread has executed buf put 17 and the consumer thread has executed
waitForBeat(1), both threads will be blocked.

The Conductor maintains a clock that wakes up periodically and checks to see if all threads
participating in the multi-threaded scenario (in this case, producer and consumer) are blocked. If so, it
increments the beat. Thus sometime later the beat will be incremented, from 0 to 1. Because consumer was
waiting for beat 1, it will wake up (i.e., the waitForBeat(1) call will return) and
execute the next line of code in its block, buf.take should be (42). This will succeed, because
the producer thread had previously (during beat 0) put 42 into the queue. This act will also make
producer runnable again, because it was blocked on the second put, which was waiting for another
thread to read that 42.

Now both threads are unblocked and able to execute their next statement. The order is
non-deterministic, and can even be simultaneous if running on multiple cores. If the consumer thread
happens to execute buf.take should be (17) first, it will block (buf.take will not return), because the queue is
at that point empty. At some point later, the producer thread will execute buf put 17, which will
unblock the consumer thread. Again both threads will be runnable and the order non-deterministic and
possibly simulataneous. The producer thread may charge ahead and run its next statement, beat should be (1).
This will succeed because the beat is indeed 1 at this point. As this is the last statement in the producer's block,
the producer thread will exit normally (it won't throw an exception). At some point later the consumer thread will
be allowed to complete its last statement, the buf.take call will return 17. The consumer thread will
execute 17 should be (17). This will succeed and as this was the last statement in its block, the consumer will return
normally.

If either the producer or consumer thread had completed abruptbly with an exception, the conduct method
(which was called by whenFinished) would have completed abruptly with an exception to indicate the test
failed. However, since both threads returned normally, conduct will return. Because conduct doesn't
throw an exception, whenFinished will execute the block of code passed as a by-name parameter to it: buf should be ('empty).
This will succeed, because the queue is indeed empty at this point. The whenFinished method will then return, and
because the whenFinished call was the last statement in the test and it didn't throw an exception, the test completes successfully.

This test tests ArrayBlockingQueue, to make sure it works as expected. If there were a bug in ArrayBlockingQueue
such as a put called on a full queue didn't block, but instead overwrote the previous value, this test would detect
it. However, if there were a bug in ArrayBlockingQueue such that a call to take called on an empty queue
never blocked and always returned 0, this test might not detect it. The reason is that whether the consumer thread will ever call
take on an empty queue during this test is non-deterministic. It depends on how the threads get scheduled during beat 1.
What is deterministic in this test, because the consumer thread blocks during beat 0, is that the producer thread will definitely
attempt to write to a full queue. To make sure the other scenario is tested, you'd need a different test:

test("calling take on an empty queue blocks the consumer thread") {

  val conductor = new Conductor
  import conductor._

  val buf = new ArrayBlockingQueue[Int](1)

  thread("producer") {
    waitForBeat(1)
    buf put 42
    buf put 17
  }

  thread("consumer") {
    buf.take should be (42)
    buf.take should be (17)
    beat should be (1)
  }

  whenFinished {
    buf should be ('empty)
  }
}

In this test, the producer thread will block, waiting for beat 1. The consumer thread will invoke buf.take
as its first act. This will block, because the queue is empty. Because both threads are blocked, the Conductor
will at some point later increment the beat to 1. This will awaken the producer thread. It will return from its
waitForBeat(1) call and execute buf put 42. This will unblock the consumer thread, which will
take the 42, and so on.

The problem that Conductor is designed to address is the difficulty, caused by the non-deterministic nature
of thread scheduling, of testing classes, traits, and libraries that are intended to be used by multiple threads.
If you just create a test in which one thread reads from an ArrayBlockingQueue and
another writes to it, you can't be sure that you have tested all possible interleavings of threads, no matter
how many times you run the test. The purpose of Conductor
is to enable you to write tests with deterministic interleavings of threads. If you write one test for each possible
interleaving of threads, then you can be sure you have all the scenarios tested. The two tests shown here, for example,
ensure that both the scenario in which a producer thread tries to write to a full queue and the scenario in which a
consumer thread tries to take from an empty queue are tested.

Class Conductor was inspired by the
MultithreadedTC project,
created by Bill Pugh and Nat Ayewah of the University of Maryland.

Although useful, bear in mind that a Conductor's results are not guaranteed to be
accurate 100% of the time. The reason is that it uses java.lang.Thread's getState method to
decide when to advance the beat. This use goes against the advice given in the Javadoc documentation for
getState, which says, "This method is designed for use in monitoring of the system state, not for
synchronization." In short, sometimes the return value of getState occasionally may be inacurrate,
which in turn means that sometimes a Conductor could decide to advance the beat too early. In practice,
Conductor has proven to be very helpful when developing thread safe classes. It is also useful in
for regression tests, but you may have to tolerate occasional false negatives.

Source

Conductors.scala

          Linear Supertypes

PatienceConfiguration, AbstractPatienceConfiguration, ScaledTimeSpans, AnyRef, Any

            Known Subclasses

ConductorFixture, ConductorMethods

<input id="mbrsel-input" type="text" accesskey="/" />

              Ordering

Alphabetic

By Inheritance

                Inherited

Conductors

PatienceConfiguration

AbstractPatienceConfiguration

ScaledTimeSpans

AnyRef

Any

Hide All

Show All

            Visibility

Public

All

Type Members

        final
        class

        Conductor extends AnyRef

Class that facilitates the testing of classes, traits, and libraries designed
to be used by multiple threads concurrently.

Class that facilitates the testing of classes, traits, and libraries designed
to be used by multiple threads concurrently.

A Conductor conducts a multi-threaded scenario by maintaining
a clock of "beats." Beats are numbered starting with 0. You can ask a
Conductor to run threads that interact with the class, trait,
or library (the subject)
you want to test. A thread can call the Conductor's
waitForBeat method, which will cause the thread to block
until that beat has been reached. The Conductor will advance
the beat only when all threads participating in the test are blocked. By
tying the timing of thread activities to specific beats, you can write
tests for concurrent systems that have deterministic interleavings of
threads.

A Conductor object has a three-phase lifecycle. It begins its life
in the setup phase. During this phase, you can start threads by
invoking the thread method on the Conductor.
When conduct is invoked on a Conductor, it enters
the conducting phase. During this phase it conducts the one multi-threaded
scenario it was designed to conduct. After all participating threads have exited, either by
returning normally or throwing an exception, the conduct method
will complete, either by returning normally or throwing an exception. As soon as
the conduct method completes, the Conductor
enters its defunct phase. Once the Conductor has conducted
a multi-threaded scenario, it is defunct and can't be reused. To run the same test again,
you'll need to create a new instance of Conductor.

Here's an example of the use of Conductor to test the ArrayBlockingQueue
class from java.util.concurrent:

import org.scalatest.fixture.FunSuite
import org.scalatest.matchers.Matchers
import java.util.concurrent.ArrayBlockingQueue
import org.scalatest.concurrent.Conductors

class ArrayBlockingQueueSuite extends FunSuite with Matchers with Conductors {

  test("calling put on a full queue blocks the producer thread") {

    val conductor = new Conductor
    import conductor._

    val buf = new ArrayBlockingQueue[Int](1)

    thread("producer") {
      buf put 42
      buf put 17
      beat should be (1)
    }

    thread("consumer") {
      waitForBeat(1)
      buf.take should be (42)
      buf.take should be (17)
    }

    whenFinished {
      buf should be ('empty)
    }
  }
}

When the test shown is run, it will create one thread named producer and another named
consumer. The producer thread will eventually execute the code passed as a by-name
parameter to thread("producer"):

buf put 42
buf put 17
beat should be (1)

Similarly, the consumer thread will eventually execute the code passed as a by-name parameter
to thread("consumer"):

waitForBeat(1)
buf.take should be (42)
buf.take should be (17)

The thread calls create the threads and starts them, but they will not immediately
execute the by-name parameter passed to them. They will first block, waiting for the Conductor
to give them a green light to proceed.

The next call in the test is whenFinished. This method will first call conduct on
the Conductor, which will wait until all threads that were created (in this case, producer and consumer) are
at the "starting line", i.e., they have all started and are blocked, waiting on the green light.
The conduct method will then give these threads the green light and they will
all start executing their blocks concurrently.

When the threads are given the green light, the beat is 0. The first thing the producer thread does is put 42 in
into the queue. As the queue is empty at this point, this succeeds. The producer thread next attempts to put a 17
into the queue, but because the queue has size 1, this can't succeed until the consumer thread has read the 42
from the queue. This hasn't happened yet, so producer blocks. Meanwhile, the consumer thread's first act is to
call waitForBeat(1). Because the beat starts out at 0, this call will block the consumer thread.
As a result, once the producer thread has executed buf put 17 and the consumer thread has executed
waitForBeat(1), both threads will be blocked.

The Conductor maintains a clock that wakes up periodically and checks to see if all threads
participating in the multi-threaded scenario (in this case, producer and consumer) are blocked. If so, it
increments the beat. Thus sometime later the beat will be incremented, from 0 to 1. Because consumer was
waiting for beat 1, it will wake up (i.e., the waitForBeat(1) call will return) and
execute the next line of code in its block, buf.take should be (42). This will succeed, because
the producer thread had previously (during beat 0) put 42 into the queue. This act will also make
producer runnable again, because it was blocked on the second put, which was waiting for another
thread to read that 42.

Now both threads are unblocked and able to execute their next statement. The order is
non-deterministic, and can even be simultaneous if running on multiple cores. If the consumer thread
happens to execute buf.take should be (17) first, it will block (buf.take will not return), because the queue is
at that point empty. At some point later, the producer thread will execute buf put 17, which will
unblock the consumer thread. Again both threads will be runnable and the order non-deterministic and
possibly simulataneous. The producer thread may charge ahead and run its next statement, beat should be (1).
This will succeed because the beat is indeed 1 at this point. As this is the last statement in the producer's block,
the producer thread will exit normally (it won't throw an exception). At some point later the consumer thread will
be allowed to complete its last statement, the buf.take call will return 17. The consumer thread will
execute 17 should be (17). This will succeed and as this was the last statement in its block, the consumer will return
normally.

If either the producer or consumer thread had completed abruptbly with an exception, the conduct method
(which was called by whenFinished) would have completed abruptly with an exception to indicate the test
failed. However, since both threads returned normally, conduct will return. Because conduct doesn't
throw an exception, whenFinished will execute the block of code passed as a by-name parameter to it: buf should be ('empty).
This will succeed, because the queue is indeed empty at this point. The whenFinished method will then return, and
because the whenFinished call was the last statement in the test and it didn't throw an exception, the test completes successfully.

This test tests ArrayBlockingQueue, to make sure it works as expected. If there were a bug in ArrayBlockingQueue
such as a put called on a full queue didn't block, but instead overwrote the previous value, this test would detect
it. However, if there were a bug in ArrayBlockingQueue such that a call to take called on an empty queue
never blocked and always returned 0, this test might not detect it. The reason is that whether the consumer thread will ever call
take on an empty queue during this test is non-deterministic. It depends on how the threads get scheduled during beat 1.
What is deterministic in this test, because the consumer thread blocks during beat 0, is that the producer thread will definitely
attempt to write to a full queue. To make sure the other scenario is tested, you'd need a different test:

test("calling take on an empty queue blocks the consumer thread") {

  val conductor = new Conductor
  import conductor._

  val buf = new ArrayBlockingQueue[Int](1)

  thread("producer") {
    waitForBeat(1)
    buf put 42
    buf put 17
  }

  thread("consumer") {
    buf.take should be (42)
    buf.take should be (17)
    beat should be (1)
  }

  whenFinished {
    buf should be ('empty)
  }
}

In this test, the producer thread will block, waiting for beat 1. The consumer thread will invoke buf.take
as its first act. This will block, because the queue is empty. Because both threads are blocked, the Conductor
will at some point later increment the beat to 1. This will awaken the producer thread. It will return from its
waitForBeat(1) call and execute buf put 42. This will unblock the consumer thread, which will
take the 42, and so on.

The problem that Conductor is designed to address is the difficulty, caused by the non-deterministic nature
of thread scheduling, of testing classes, traits, and libraries that are intended to be used by multiple threads.
If you just create a test in which one thread reads from an ArrayBlockingQueue and
another writes to it, you can't be sure that you have tested all possible interleavings of threads, no matter
how many times you run the test. The purpose of Conductor
is to enable you to write tests with deterministic interleavings of threads. If you write one test for each possible
interleaving of threads, then you can be sure you have all the scenarios tested. The two tests shown here, for example,
ensure that both the scenario in which a producer thread tries to write to a full queue and the scenario in which a
consumer thread tries to take from an empty queue are tested.

Class Conductor was inspired by the
MultithreadedTC project,
created by Bill Pugh and Nat Ayewah of the University of Maryland.

Although useful, bear in mind that a Conductor's results are not guaranteed to be
accurate 100% of the time. The reason is that it uses java.lang.Thread's getState method to
decide when to advance the beat. This use goes against the advice given in the Javadoc documentation for
getState, which says, "This method is designed for use in monitoring of the system state, not for
synchronization." In short, sometimes the return value of getState occasionally may be inacurrate,
which in turn means that sometimes a Conductor could decide to advance the beat too early. In practice,
Conductor has proven to be very helpful when developing thread safe classes. It is also useful in
for regression tests, but you may have to tolerate occasional false negatives.

        final
        case class

        PatienceConfig(timeout: Span = scaled(Span(150, Millis)), interval: Span = scaled(Span(15, Millis))) extends Product with Serializable

Configuration object for asynchronous constructs, such as those provided by traits Eventually and
Waiters.

Configuration object for asynchronous constructs, such as those provided by traits Eventually and
Waiters.

The default values for the parameters are:

Configuration Parameter	Default Value

timeout

scaled(150 milliseconds)

interval

scaled(15 milliseconds)

timeout

the maximum amount of time to wait for an asynchronous operation to complete before giving up and throwing
  TestFailedException.

interval

the amount of time to sleep between each check of the status of an asynchronous operation when polling

Definition Classes

AbstractPatienceConfiguration

Value Members

        final
        def

        !=(arg0: Any): Boolean

Definition Classes

AnyRef → Any

        final
        def

        ##(): Int

Definition Classes

AnyRef → Any

        final
        def

        ==(arg0: Any): Boolean

Definition Classes

AnyRef → Any

        final
        def

        asInstanceOf[T0]: T0

Definition Classes

Any

        def

        clone(): AnyRef

Attributes

protected[java.lang]

Definition Classes

AnyRef

Annotations

                @throws(

      ...
    )

        final
        def

        eq(arg0: AnyRef): Boolean

Definition Classes

AnyRef

        def

        equals(arg0: Any): Boolean

Definition Classes

AnyRef → Any

        def

        finalize(): Unit

Attributes

protected[java.lang]

Definition Classes

AnyRef

Annotations

                @throws(

      classOf[java.lang.Throwable]
    )

        final
        def

        getClass(): Class[_]

Definition Classes

AnyRef → Any

        def

        hashCode(): Int

Definition Classes

AnyRef → Any

        def

        interval(value: Span): Interval

Returns an Interval configuration parameter containing the passed value, which
specifies the amount of time to sleep after a retry.

Returns an Interval configuration parameter containing the passed value, which
specifies the amount of time to sleep after a retry.

Definition Classes

PatienceConfiguration

        final
        def

        isInstanceOf[T0]: Boolean

Definition Classes

Any

        final
        def

        ne(arg0: AnyRef): Boolean

Definition Classes

AnyRef

        final
        def

        notify(): Unit

Definition Classes

AnyRef

        final
        def

        notifyAll(): Unit

Definition Classes

AnyRef

        implicit
        def

        patienceConfig: PatienceConfig

Implicit PatienceConfig value providing default configuration values.

Implicit PatienceConfig value providing default configuration values.

To change the default configuration, override or hide this def with another implicit
PatienceConfig containing your desired default configuration values.

Definition Classes

PatienceConfiguration → AbstractPatienceConfiguration

        final
        def

        scaled(span: Span): Span

Scales the passed Span by the Double factor returned
by spanScaleFactor.

Scales the passed Span by the Double factor returned
by spanScaleFactor.

The Span is scaled by invoking its scaledBy method,
thus this method has the same behavior:
The value returned by spanScaleFactor can be any positive number or zero,
including a fractional number. A number greater than one will scale the Span
up to a larger value. A fractional number will scale it down to a smaller value. A
factor of 1.0 will cause the exact same Span to be returned. A
factor of zero will cause Span.ZeroLength to be returned.
If overflow occurs, Span.Max will be returned. If underflow occurs,
Span.ZeroLength will be returned.

Definition Classes

ScaledTimeSpans

Exceptions thrown

IllegalArgumentException if the value returned from spanScaleFactor
          is less than zero

        def

        spanScaleFactor: Double

The factor by which the scaled method will scale Spans.

The factor by which the scaled method will scale Spans.

The default implementation of this method will return the span scale factor that
was specified for the run, or 1.0 if no factor was specified. For example, you can specify a span scale factor when invoking ScalaTest
via the command line by passing a -F argument to Runner.

Definition Classes

ScaledTimeSpans

        final
        def

        synchronized[T0](arg0: ⇒ T0): T0

Definition Classes

AnyRef

        def

        timeout(value: Span): Timeout

Returns a Timeout configuration parameter containing the passed value, which
specifies the maximum amount to wait for an asynchronous operation to complete.

Returns a Timeout configuration parameter containing the passed value, which
specifies the maximum amount to wait for an asynchronous operation to complete.

Definition Classes

PatienceConfiguration

        def

        toString(): String

Definition Classes

AnyRef → Any

        final
        def

        wait(): Unit

Definition Classes

AnyRef

Annotations

                @throws(

      ...
    )

        final
        def

        wait(arg0: Long, arg1: Int): Unit

Definition Classes

AnyRef

Annotations

                @throws(

      ...
    )

        final
        def

        wait(arg0: Long): Unit

Definition Classes

AnyRef

Annotations

                @throws(

      ...
    )

Inherited from PatienceConfiguration

Inherited from AbstractPatienceConfiguration

Inherited from ScaledTimeSpans

Inherited from AnyRef

Inherited from Any

Ungrouped


text:
<?php

use Doctrine\ORM\EntityManager;
use DvsaCommon\Factory\AutoWire\AutoWireFactory;
use DvsaCommon\HttpRestJson\Client;
use DvsaCommon\HttpRestJson\ZendClient;
use DvsaCommon\Obfuscate\ParamEncrypter;
use DvsaCommon\Obfuscate\ParamObfuscator;
use DvsaCommon\FullStackLogging\Service\FullStackLoggingService;
use DvsaCommon\FullStackLogging\Provider\TraceIdProvider;
use DvsaCommon\FullStackLogging\Provider\SpanIdProvider;
use DvsaCommon\FullStackLogging\Provider\ParentSpanIdProvider;
use TestSupport\Factory\AnnualAssessmentAdministratorServiceFactory;
use TestSupport\Factory\CatUserServiceFactory;
use TestSupport\Factory\ClaimAccountServiceFactory;
use TestSupport\Factory\GracePeriodServiceFactory;
use TestSupport\Factory\StatisticsDbServiceFactory;
use TestSupport\Factory\TesterAuthorisationStatusServiceFactory;
use TestSupport\Factory\UserInboxServiceFactory;
use TestSupport\Helper\NotificationsHelper;
use TestSupport\Helper\SitePermissionsHelper;
use TestSupport\Helper\TestSupportAccessTokenManager;
use TestSupport\Helper\TestSupportRestClientHelper;
use TestSupport\Service\AccountDataService;
use TestSupport\Service\AccountService;
use TestSupport\Service\AeUserService;
use TestSupport\Service\AEService;
use TestSupport\Service\AnnualAssessmentAdministratorService;
use TestSupport\Service\AreaOffice1Service;
use TestSupport\Service\CatUserService;
use TestSupport\Service\CertificateReplacementService;
use TestSupport\Service\ClaimAccountService;
use TestSupport\Service\CronUserService;
use TestSupport\Service\CSCOService;
use TestSupport\Service\CSMService;
use TestSupport\Service\DocumentService;
use TestSupport\Service\DVLAManagerService;
use TestSupport\Service\DVLAOperativeService;
use TestSupport\Service\DvlaVehicleService;
use TestSupport\Service\EventService;
use TestSupport\Service\FeaturesService;
use TestSupport\Service\FinanceUserService;
use TestSupport\Service\GracePeriodService;
use TestSupport\Service\GVTSTesterService;
use TestSupport\Service\InactiveTesterService;
use TestSupport\Service\JsonErrorHandlingListener;
use TestSupport\Service\MotService;
use TestSupport\Service\PasswordResetService;
use TestSupport\Service\PaymentNotificationsAuditService;
use TestSupport\Service\SchemeManagerService;
use TestSupport\Service\SchemeUserService;
use TestSupport\Service\SiteRoleNominationService;
use TestSupport\Service\SiteUserDataService;
use TestSupport\Service\SlotTransactionService;
use TestSupport\Service\StatisticsAmazonCacheService;
use TestSupport\Service\StatisticsDbService;
use TestSupport\Service\TesterAuthorisationStatusService;
use TestSupport\Service\TesterService;
use TestSupport\Service\UserInboxService;
use TestSupport\Service\UserService;
use TestSupport\Service\VehicleExaminerService;
use TestSupport\Service\VehicleService;
use TestSupport\Service\VM10519UserService;
use TestSupport\Service\VM10619RoleManagementUpgradeService;
use TestSupport\Service\TwoFactorAuthCardService;
use TestSupport\Service\VtsService;
use Zend\Http\Request;
use Zend\Console\Request as ConsoleRequest;
use Zend\Log\Logger;
use Zend\ServiceManager\ServiceLocatorInterface;
use TestSupport\Service\GdsSurveyService;
use TestSupport\Factory\GdsSurveyServiceFactory;
use TestSupport\Service\SecurityQuestionsService;
use TestSupport\Factory\SecurityQuestionsServiceFactory;
use TestSupport\Factory\SiteRoleNominationServiceFactory;
use TestSupport\Service\OrganisationRoleNominationService;
use TestSupport\Factory\RoleNominationServiceFactory;
use TestSupport\Service\VehicleTestingAdviceService;

return [
    'factories' => [
        TestSupportAccessTokenManager::class => function (ServiceLocatorInterface $sm) {
            $config = $sm->get('config');

            return new TestSupportAccessTokenManager($config['apiUrl']);
        },
        JsonErrorHandlingListener::class => function (ServiceLocatorInterface $sm) {
            return new JsonErrorHandlingListener();
        },

        FullStackLoggingService::class => function (ServiceLocatorInterface $sm){
            $request = $sm->get('Request');

            // we only deal with Zend\Http\Request here (for http headers support)
            if($request instanceof ConsoleRequest) {
                $request = new Request();
            }

            $stackName = 'TestSupport';
            $loggerStub =  new Logger(
                [
                    'writers' => [
                        'noop' => [
                            'adapter' => '\Zend\Log\Writer\Noop',
                            'name' => 'noop',
                        ],
                    ],
                ]
            );

            return new FullStackLoggingService(
                $request,
                new TraceIdProvider($stackName),
                new SpanIdProvider($stackName),
                new ParentSpanIdProvider($stackName),
                $loggerStub
            );
        },

        \DvsaCommon\HttpRestJson\Client::class => function (ServiceLocatorInterface $sm) {
            $config = $sm->get('config');
            /** @var FullStackLoggingService $fullStackLoggingService */
            $fullStackLoggingService = $sm->get(FullStackLoggingService::class);

            return new ZendClient(
                new \Zend\Http\Client(),
                $config['apiUrl'],
                $fullStackLoggingService
            );
        },
        SiteUserDataService::class => function (ServiceLocatorInterface $sm) {
            return new SiteUserDataService(
                    $sm->get(NotificationsHelper::class),
                    $sm->get(SitePermissionsHelper::class),
                    $sm->get(AccountService::class)
                );
        },
        AccountDataService::class => \TestSupport\Factory\AccountDataServiceFactory::class,
        AccountService::class => function (ServiceLocatorInterface $sm) {
            $tokenManager = $sm->get(TestSupportAccessTokenManager::class);

            return new AccountService(
                    $sm->get(EntityManager::class),
                    $sm->get(Client::class),
                    $tokenManager,
                    $sm->get(SecurityQuestionsService::class)
                );
        },
        VehicleService::class => function (ServiceLocatorInterface $sm) {
            $entityManager = $sm->get(EntityManager::class);
            $tokenManager = $sm->get(TestSupportAccessTokenManager::class);
            /** @var FullStackLoggingService $fullStackLoggingService */
            $fullStackLoggingService = $sm->get(FullStackLoggingService::class);
            $config = $sm->get('config');
            $vehicleServiceUrl = isset($config['api']) && isset($config['api']['vehicle_service_url']) ?
                    $config['api']['vehicle_service_url'] :
                    null;

            return new VehicleService(
                    $entityManager,
                    $tokenManager,
                    $vehicleServiceUrl,
                    $fullStackLoggingService
                );
        },
        DvlaVehicleService::class => function (ServiceLocatorInterface $sm) {
            return new DvlaVehicleService($sm->get(EntityManager::class));
        },
        EventService::class => function (ServiceLocatorInterface $sm) {
            return new EventService($sm->get(EntityManager::class));
        },
        VehicleTestingAdviceService::class => function (ServiceLocatorInterface $sm) {
            return new VehicleTestingAdviceService($sm->get(EntityManager::class));
        },
        PaymentNotificationsAuditService::class => function (ServiceLocatorInterface $sm) {
            return new PaymentNotificationsAuditService($sm->get(EntityManager::class));
        },
        SlotTransactionService::class => function (ServiceLocatorInterface $sm) {
            $tokenManager = $sm->get(TestSupportAccessTokenManager::class);

            return new SlotTransactionService($sm->get(Client::class), $tokenManager);
        },
        MotService::class => function (ServiceLocatorInterface $sm) {
            return new MotService($sm->get(EntityManager::class));
        },
        CertificateReplacementService::class => function (ServiceLocatorInterface $sm) {
            return new CertificateReplacementService($sm->get(EntityManager::class));
        },

        // @TODO after mot-common-web-module is part of composer remove the below lines as the module will
        // already have these services registered.
        ParamEncrypter::class => \DvsaCommon\Obfuscate\Factory\ParamEncrypterFactory::class,
        ParamObfuscator::class => \DvsaCommon\Obfuscate\Factory\ParamObfuscatorFactory::class,
        'ApplicationLog' => 'TestSupport\Factory\ApplicationLogFactory',
        CSCOService::class => \TestSupport\Factory\CSCOServiceFactory::class,
        CSMService::class => \TestSupport\Factory\CSMServiceFactory::class,
        DVLAManagerService::class => \TestSupport\Factory\DVLAManagerServiceFactory::class,
        DVLAOperativeService::class => \TestSupport\Factory\DVLAOperativeServiceFactory::class,
        AreaOffice1Service::class => \TestSupport\Factory\AreaOffice1ServiceFactory::class,
        CronUserService::class => \TestSupport\Factory\CronUserServiceFactory::class,
        FinanceUserService::class => \TestSupport\Factory\FinanceUserServiceFactory::class,
        UserService::class => \TestSupport\Factory\UserServiceFactory::class,
        VtsService::class => \TestSupport\Factory\VtsServiceFactory::class,
        TestSupportRestClientHelper::class => \TestSupport\Factory\TestSupportRestClientFactory::class,
        NotificationsHelper::class => \TestSupport\Factory\NotificationsHelperFactory::class,
        SitePermissionsHelper::class => \TestSupport\Factory\SitePermissionsHelperFactory::class,
        AEService::class => \TestSupport\Factory\AEServiceFactory::class,
        TesterService::class => \TestSupport\Factory\TesterServiceFactory::class,
        FeaturesService::class => \TestSupport\Factory\FeaturesServiceFactory::class,
        InactiveTesterService::class => \TestSupport\Factory\InactiveTesterServiceFactory::class,
        SchemeManagerService::class => \TestSupport\Factory\SchemeManagerServiceFactory::class,
        SchemeUserService::class => \TestSupport\Factory\SchemeUserServiceFactory::class,
        InactiveTesterService::class => \TestSupport\Factory\InactiveTesterServiceFactory::class,
        PasswordResetService::class => \TestSupport\Factory\PasswordResetServiceFactory::class,
        VehicleExaminerService::class => \TestSupport\Factory\VehicleExaminerServiceFactory::class,
        VM10519UserService::class => \TestSupport\Factory\VM10519UserServiceFactory::class,
        VM10619RoleManagementUpgradeService::class => \TestSupport\Factory\VM10619RoleManagementUpgradeServiceFactory::class,
        TesterAuthorisationStatusService::class => TesterAuthorisationStatusServiceFactory::class,
        TwoFactorAuthCardService::class => \TestSupport\Factory\TwoFactorAuthCardServiceFactory::class,
        AeUserService::class => \TestSupport\Factory\AedmServiceFactory::class,
        DocumentService::class => \TestSupport\Factory\DocumentServiceFactory::class,
        GVTSTesterService::class => \TestSupport\Factory\GVTSTesterServiceFactory::class,
        StatisticsAmazonCacheService::class => \TestSupport\Factory\StatisticsAmazonCacheFactory::class,
        StatisticsDbService::class => StatisticsDbServiceFactory::class,
        GdsSurveyService::class => GdsSurveyServiceFactory::class,
        SecurityQuestionsService::class => SecurityQuestionsServiceFactory::class,
        ClaimAccountService::class => ClaimAccountServiceFactory::class,
        CatUserService::class => CatUserServiceFactory::class,
        SiteRoleNominationService::class => SiteRoleNominationServiceFactory::class,
        OrganisationRoleNominationService::class => RoleNominationServiceFactory::class,
        AnnualAssessmentAdministratorService::class => AnnualAssessmentAdministratorServiceFactory::class,
        \TestSupport\Service\QualificationCertificateService::class => \TestSupport\Factory\QualificationCertificateServiceFactory::class,
        GracePeriodService::class => GracePeriodServiceFactory::class,
    ],
    'abstract_factories' => [
        AutoWireFactory::class,
    ],
];

redpajama_set_name: RedPajamaGithub


text:
r"""Pipeline for loading VCF files to BigQuery.

Run locally:
python -m gcp_variant_transforms.vcf_to_bq \
  --input_pattern <path to VCF file(s)> \
  --output_table projectname:bigquerydataset.tablename

Run on Dataflow:
python -m gcp_variant_transforms.vcf_to_bq \
  --input_pattern gs://bucket/vcfs/vcffile.vcf \
  --output_table projectname:bigquerydataset.tablename \
  --project projectname \
  --staging_location gs://bucket/staging \
  --temp_location gs://bucket/temp \
  --job_name vcf-to-bq \
  --setup_file ./setup.py \
  --runner DataflowRunner
"""

import argparse  # pylint: disable=unused-import
from datetime import datetime
import logging
import sys
import tempfile
from typing import List, Optional  # pylint: disable=unused-import

import apache_beam as beam
from apache_beam import pvalue  # pylint: disable=unused-import
from apache_beam.io import filesystems
from apache_beam.options import pipeline_options

from gcp_variant_transforms import pipeline_common
from gcp_variant_transforms.beam_io import vcf_parser
from gcp_variant_transforms.libs import avro_util
from gcp_variant_transforms.libs import bigquery_util
from gcp_variant_transforms.libs import metrics_util
from gcp_variant_transforms.libs import partitioning
from gcp_variant_transforms.libs import processed_variant
from gcp_variant_transforms.libs import sample_info_table_schema_generator
from gcp_variant_transforms.libs import schema_converter
from gcp_variant_transforms.libs import vcf_header_parser
from gcp_variant_transforms.libs import variant_sharding
from gcp_variant_transforms.libs.annotation.vep import vep_runner_util
from gcp_variant_transforms.libs.variant_merge import merge_with_non_variants_strategy
from gcp_variant_transforms.libs.variant_merge import move_to_calls_strategy
from gcp_variant_transforms.libs.variant_merge import variant_merge_strategy  # pylint: disable=unused-import
from gcp_variant_transforms.options import variant_transform_options
from gcp_variant_transforms.transforms import annotate_files
from gcp_variant_transforms.transforms import sample_info_to_avro
from gcp_variant_transforms.transforms import combine_sample_ids
from gcp_variant_transforms.transforms import densify_variants
from gcp_variant_transforms.transforms import extract_input_size
from gcp_variant_transforms.transforms import filter_variants
from gcp_variant_transforms.transforms import infer_headers
from gcp_variant_transforms.transforms import merge_headers
from gcp_variant_transforms.transforms import merge_variants
from gcp_variant_transforms.transforms import shard_variants
from gcp_variant_transforms.transforms import variant_to_avro
from gcp_variant_transforms.transforms import write_variants_to_shards

_COMMAND_LINE_OPTIONS = [
    variant_transform_options.VcfReadOptions,
    variant_transform_options.BigQueryWriteOptions,
    variant_transform_options.AnnotationOptions,
    variant_transform_options.FilterOptions,
    variant_transform_options.MergeOptions,
    variant_transform_options.ExperimentalOptions,
]

_ESTIMATE_SIZES_FILE_NAME = 'estimate-sizes'
_ESTIMATE_SIZES_JOB_NAME = 'estimate-input-size'
_MERGE_HEADERS_FILE_NAME = 'merged_headers.vcf'
_MERGE_HEADERS_JOB_NAME = 'merge-vcf-headers'
_ANNOTATE_FILES_JOB_NAME = 'annotate-files'
_SHARD_VCF_FILES_JOB_NAME = 'shard-files'
_BQ_SCHEMA_FILE_SUFFIX = 'schema.json'
_AVRO_FOLDER = 'avro'
_SHARDS_FOLDER = 'shards'
_GCS_RECURSIVE_WILDCARD = '**'
SampleNameEncoding = vcf_parser.SampleNameEncoding

_newly_created_tables = []
def _record_newly_created_table(full_table_id):
  global _newly_created_tables  # pylint: disable=global-statement
  _newly_created_tables.append(full_table_id)

_new_temp_tables = []
def _record_new_temp_table(full_table_id):
  global _new_temp_tables  # pylint: disable=global-statement
  _new_temp_tables.append(full_table_id)

def _read_variants(all_patterns,  # type: List[str]
                   pipeline,  # type: beam.Pipeline
                   known_args,  # type: argparse.Namespace
                   pipeline_mode,  # type: int
                   pre_infer_headers=False,  # type: bool
                   keep_raw_sample_names=False,  # type: bool
                   use_1_based_coordinate=True,  # type: bool
                   move_hom_ref_calls=False  # type: bool
                  ):
  # type: (...) -> pvalue.PCollection
  """Helper method for returning a PCollection of Variants from VCFs."""
  representative_header_lines = None
  if known_args.representative_header_file:
    representative_header_lines = vcf_header_parser.get_metadata_header_lines(
        known_args.representative_header_file)
  return pipeline_common.read_variants(
      pipeline,
      all_patterns,
      pipeline_mode,
      known_args.allow_malformed_records,
      representative_header_lines,
      pre_infer_headers=pre_infer_headers,
      sample_name_encoding=(
          SampleNameEncoding.NONE if keep_raw_sample_names else
          SampleNameEncoding[known_args.sample_name_encoding]),
      use_1_based_coordinate=use_1_based_coordinate,
      move_hom_ref_calls=move_hom_ref_calls)

def _get_variant_merge_strategy(known_args  # type: argparse.Namespace
                               ):
  # type: (...) -> Optional[variant_merge_strategy.VariantMergeStrategy]
  merge_options = variant_transform_options.MergeOptions
  if (not known_args.variant_merge_strategy or
      known_args.variant_merge_strategy == merge_options.NONE):
    return None
  elif known_args.variant_merge_strategy == merge_options.MOVE_TO_CALLS:
    return move_to_calls_strategy.MoveToCallsStrategy(
        known_args.info_keys_to_move_to_calls_regex,
        known_args.copy_quality_to_calls,
        known_args.copy_filter_to_calls)
  elif (known_args.variant_merge_strategy ==
        merge_options.MERGE_WITH_NON_VARIANTS):
    return merge_with_non_variants_strategy.MergeWithNonVariantsStrategy(
        known_args.info_keys_to_move_to_calls_regex,
        known_args.copy_quality_to_calls,
        known_args.copy_filter_to_calls)
  else:
    raise ValueError('Merge strategy is not supported.')

def _add_inferred_headers(all_patterns,  # type: List[str]
                          pipeline,  # type: beam.Pipeline
                          known_args,  # type: argparse.Namespace
                          merged_header,  # type: pvalue.PCollection
                          pipeline_mode  # type: int
                         ):
  # type: (...) -> pvalue.PCollection
  annotation_fields_to_infer = (known_args.annotation_fields if
                                known_args.infer_annotation_types else [])
  inferred_headers = (
      _read_variants(all_patterns,
                     pipeline,
                     known_args,
                     pipeline_mode,
                     pre_infer_headers=known_args.infer_headers)
      | 'FilterVariants' >> filter_variants.FilterVariants(
          reference_names=known_args.reference_names)
      | 'InferHeaderFields' >> infer_headers.InferHeaderFields(
          pvalue.AsSingleton(merged_header),
          known_args.allow_incompatible_records,
          known_args.infer_headers,
          annotation_fields_to_infer))
  merged_header = (
      (inferred_headers, merged_header)
      | 'FlattenHeaders' >> beam.Flatten()
      | 'MergeHeadersFromVcfAndVariants' >> merge_headers.MergeHeaders(
          known_args.split_alternate_allele_info_fields,
          known_args.allow_incompatible_records))
  return merged_header

def _shard_variants(known_args, pipeline_args, pipeline_mode):
  # type: (argparse.Namespace, List[str], int) -> List[str]
  """Reads the variants and writes them to VCF shards.

  Returns:
   The VCF shards directory.
  """
  options = pipeline_options.PipelineOptions(pipeline_args)
  google_cloud_options = options.view_as(pipeline_options.GoogleCloudOptions)
  shard_files_job_name = pipeline_common.generate_unique_name(
      _SHARD_VCF_FILES_JOB_NAME)
  _update_google_cloud_job_name(google_cloud_options, shard_files_job_name)
  vcf_shards_output_dir = filesystems.FileSystems.join(
      known_args.annotation_output_dir, _SHARDS_FOLDER)
  with beam.Pipeline(options=options) as p:
    variants = _read_variants(known_args.all_patterns,
                              p,
                              known_args,
                              pipeline_mode,
                              pre_infer_headers=False,
                              keep_raw_sample_names=True,
                              use_1_based_coordinate=False,
                              move_hom_ref_calls=False)
    sample_ids = (variants
                  | 'CombineSampleIds' >>
                  combine_sample_ids.SampleIdsCombiner()
                  | 'CombineToList' >> beam.combiners.ToList())
    # TODO(tneymanov): Annotation pipeline currently stores sample IDs instead
    # of sample names in the the sharded VCF files, which would lead to double
    # hashing of samples. Needs to be fixed ASAP.
    _ = (variants
         | 'DensifyVariants' >> densify_variants.DensifyVariants(
             beam.pvalue.AsSingleton(sample_ids))
         | 'WriteToShards' >> write_variants_to_shards.WriteToShards(
             vcf_shards_output_dir,
             beam.pvalue.AsSingleton(sample_ids),
             known_args.number_of_variants_per_shard))

  return [vep_runner_util.format_dir_path(vcf_shards_output_dir) +
          _GCS_RECURSIVE_WILDCARD]

def _get_input_dimensions(known_args, pipeline_args):
  pipeline_mode = pipeline_common.get_pipeline_mode(known_args.all_patterns, known_args.pipeline_mode)
  beam_pipeline_options = pipeline_options.PipelineOptions(pipeline_args)
  google_cloud_options = beam_pipeline_options.view_as(
      pipeline_options.GoogleCloudOptions)

  estimate_sizes_job_name = pipeline_common.generate_unique_name(
      _ESTIMATE_SIZES_JOB_NAME)
  if google_cloud_options.job_name:
    google_cloud_options.job_name += '-' + estimate_sizes_job_name
  else:
    google_cloud_options.job_name = estimate_sizes_job_name
  temp_directory = google_cloud_options.temp_location or tempfile.mkdtemp()
  temp_estimated_input_size_file_name = '-'.join(
      [google_cloud_options.job_name,
       _ESTIMATE_SIZES_FILE_NAME])
  temp_estimated_input_size_file_path = filesystems.FileSystems.join(
      temp_directory, temp_estimated_input_size_file_name)
  with beam.Pipeline(options=beam_pipeline_options) as p:
    estimates = pipeline_common.get_estimates(
        p, pipeline_mode, known_args.all_patterns)

    files_size = (estimates
                  | 'GetFilesSize' >> extract_input_size.GetFilesSize())
    file_count = (estimates
                  | 'CountAllFiles' >> beam.combiners.Count.Globally())
    sample_map = (estimates
                  | 'ExtractSampleMap' >> extract_input_size.GetSampleMap())
    estimated_value_count = (sample_map
                             | extract_input_size.GetEstimatedValueCount())
    estimated_sample_count = (sample_map
                              | extract_input_size.GetEstimatedSampleCount())
    estimated_variant_count = (estimates
                               | 'GetEstimatedVariantCount'
                               >> extract_input_size.GetEstimatedVariantCount())
    _ = (estimated_variant_count
         | beam.ParDo(extract_input_size.print_estimates_to_file,
                      beam.pvalue.AsSingleton(estimated_sample_count),
                      beam.pvalue.AsSingleton(estimated_value_count),
                      beam.pvalue.AsSingleton(files_size),
                      beam.pvalue.AsSingleton(file_count),
                      temp_estimated_input_size_file_path))

  with filesystems.FileSystems.open(temp_estimated_input_size_file_path) as f:
    estimates = f.readlines()
  if len(estimates) != 5:
    raise ValueError('Exactly 5 estimates were expected in {}.'.format(
        temp_estimated_input_size_file_path))

  known_args.estimated_variant_count = int(estimates[0].strip())
  known_args.estimated_sample_count = int(estimates[1].strip())
  known_args.estimated_value_count = int(estimates[2].strip())
  known_args.files_size = int(estimates[3].strip())
  known_args.file_count = int(estimates[4].strip())

def _annotate_vcf_files(all_patterns, known_args, pipeline_args):
  # type: (List[str], argparse.Namespace, List[str]) -> str
  """Annotates the VCF files using VEP.

  Returns:
    The annotated VCF files directory.
  """
  options = pipeline_options.PipelineOptions(pipeline_args)
  google_cloud_options = options.view_as(pipeline_options.GoogleCloudOptions)
  annotate_files_job_name = pipeline_common.generate_unique_name(
      _ANNOTATE_FILES_JOB_NAME)
  _update_google_cloud_job_name(google_cloud_options, annotate_files_job_name)

  with beam.Pipeline(options=options) as p:
    _ = (p
         | beam.Create(all_patterns)
         | 'AnnotateShards' >> beam.ParDo(
             annotate_files.AnnotateFile(known_args, pipeline_args)))
  if known_args.annotation_fields:
    known_args.annotation_fields.append(known_args.vep_info_field)
  else:
    known_args.annotation_fields = [known_args.vep_info_field]
  # TODO(bashir2): The VEP runner by default runs VEP with --allele_number hence
  # we turn on this feature here. However, this might be inconsistent with other
  # annotation fields that are originally present in input files, if they do not
  # have ALLELE_NUM annotation. The fix is to make annotation ALT matching
  # smarter to fall back on other matching methods if ALLELE_NUM is not present.
  # When this is implemented, we may even consider removing use_allele_num flag
  # and always start by checking if ALLELE_NUM is present.
  known_args.use_allele_num = True
  return vep_runner_util.get_output_pattern(known_args.annotation_output_dir)

def _update_google_cloud_job_name(google_cloud_options, job_name):
  if google_cloud_options.job_name:
    google_cloud_options.job_name += '-' + job_name
  else:
    google_cloud_options.job_name = job_name

def _merge_headers(known_args, pipeline_args,
                   pipeline_mode, avro_root_path, annotated_vcf_pattern=None):
  # type: (str, argparse.Namespace, List[str], int, str) -> None
  """Merges VCF headers using beam based on pipeline_mode."""
  options = pipeline_options.PipelineOptions(pipeline_args)

  # Always run pipeline locally if data is small.
  if (pipeline_mode == pipeline_common.PipelineModes.SMALL and
      not known_args.infer_headers and not known_args.infer_annotation_types):
    options.view_as(pipeline_options.StandardOptions).runner = 'DirectRunner'

  google_cloud_options = options.view_as(pipeline_options.GoogleCloudOptions)
  merge_headers_job_name = pipeline_common.generate_unique_name(
      _MERGE_HEADERS_JOB_NAME)
  if google_cloud_options.job_name:
    google_cloud_options.job_name += '-' + merge_headers_job_name
  else:
    google_cloud_options.job_name = merge_headers_job_name

  temp_directory = google_cloud_options.temp_location or tempfile.mkdtemp()
  temp_merged_headers_file_name = '-'.join([google_cloud_options.job_name,
                                            _MERGE_HEADERS_FILE_NAME])
  temp_merged_headers_file_path = filesystems.FileSystems.join(
      temp_directory, temp_merged_headers_file_name)

  with beam.Pipeline(options=options) as p:
    headers = pipeline_common.read_headers(
        p, pipeline_mode,
        known_args.all_patterns)
    _ = (headers
         | 'SampleInfoToAvro'
         >> sample_info_to_avro.SampleInfoToAvro(
             avro_root_path +
             sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX,
             SampleNameEncoding[known_args.sample_name_encoding]))
    if known_args.representative_header_file:
      return
    merged_header = pipeline_common.get_merged_headers(
        headers,
        known_args.split_alternate_allele_info_fields,
        known_args.allow_incompatible_records)
    if annotated_vcf_pattern:
      merged_header = pipeline_common.add_annotation_headers(
          p, known_args, pipeline_mode, merged_header,
          annotated_vcf_pattern)
    if known_args.infer_headers or known_args.infer_annotation_types:
      infer_headers_input_pattern = (
          [annotated_vcf_pattern] if
          annotated_vcf_pattern else known_args.all_patterns)
      merged_header = _add_inferred_headers(infer_headers_input_pattern, p,
                                            known_args, merged_header,
                                            pipeline_mode)

    pipeline_common.write_headers(merged_header, temp_merged_headers_file_path)
    known_args.representative_header_file = temp_merged_headers_file_path

def _validate_annotation_pipeline_args(known_args, pipeline_args):
  match_results = filesystems.FileSystems.match(['{}*'.format(
      vep_runner_util.format_dir_path(known_args.annotation_output_dir))])
  if match_results and match_results[0].metadata_list:
    raise ValueError('Output directory {} already exists.'.format(
        known_args.annotation_output_dir))

  flags_dict = pipeline_options.PipelineOptions(pipeline_args).get_all_options()
  expected_flags = ['max_num_workers', 'num_workers']
  for flag in expected_flags:
    if (flag in flags_dict and
        flags_dict[flag] is not None and flags_dict[flag] > 0):
      return
  raise ValueError('Could not find any of {} with a valid value among pipeline '
                   'flags {}'.format(expected_flags, flags_dict))

def _run_annotation_pipeline(known_args, pipeline_args):
  # type: (argparse.Namespace, List[str]) -> str
  annotated_vcf_pattern = None
  if known_args.run_annotation_pipeline:
    _validate_annotation_pipeline_args(known_args, pipeline_args)
    known_args.omit_empty_sample_calls = True

    files_to_be_annotated = known_args.all_patterns
    if known_args.shard_variants:
      pipeline_mode = pipeline_common.get_pipeline_mode(files_to_be_annotated, known_args.pipeline_mode)
      files_to_be_annotated = _shard_variants(known_args,
                                              pipeline_args,
                                              pipeline_mode)
    annotated_vcf_pattern = _annotate_vcf_files(files_to_be_annotated,
                                                known_args,
                                                pipeline_args)
  return annotated_vcf_pattern

def _write_schema_to_temp_file(schema, path):
  if not path or not path.startswith('gs://'):
    raise ValueError('Schema must be stored on a GS bucket.')
  schema_json = schema_converter.convert_table_schema_to_json_bq_schema(schema)
  schema_file = tempfile.mkstemp(suffix=_BQ_SCHEMA_FILE_SUFFIX)[1]
  with filesystems.FileSystems.create(schema_file) as file_to_write:
    file_to_write.write(schema_json.encode('utf-8'))
  gs_file_path = path + _BQ_SCHEMA_FILE_SUFFIX
  with filesystems.FileSystems.create(gs_file_path) as file_to_write:
    file_to_write.write(schema_json.encode('utf-8'))
  return schema_file

def _get_avro_root_path(beam_pipeline_options):
  google_cloud_options = beam_pipeline_options.view_as(
      pipeline_options.GoogleCloudOptions)
  return filesystems.FileSystems.join(google_cloud_options.temp_location,
                                      _AVRO_FOLDER,
                                      google_cloud_options.job_name,
                                      datetime.now().strftime('%Y%m%d_%H%M%S'),
                                      '')

def run(argv=None):
  # type: (List[str]) -> None
  """Runs VCF to BigQuery pipeline."""
  logging.info('Command: %s', ' '.join(argv or sys.argv))
  known_args, pipeline_args = pipeline_common.parse_args(argv,
                                                         _COMMAND_LINE_OPTIONS)

  if known_args.auto_flags_experiment:
    _get_input_dimensions(known_args, pipeline_args)

  annotated_vcf_pattern = _run_annotation_pipeline(known_args, pipeline_args)

  all_patterns = (
      [annotated_vcf_pattern] if annotated_vcf_pattern
      else known_args.all_patterns)

  variant_merger = _get_variant_merge_strategy(known_args)

  pipeline_mode = pipeline_common.get_pipeline_mode(all_patterns, known_args.pipeline_mode)

  beam_pipeline_options = pipeline_options.PipelineOptions(pipeline_args)
  avro_root_path = _get_avro_root_path(beam_pipeline_options)
  # Starts a pipeline to merge VCF headers in beam if the total files that
  # match the input pattern exceeds _SMALL_DATA_THRESHOLD
  _merge_headers(known_args, pipeline_args,
                 pipeline_mode, avro_root_path, annotated_vcf_pattern)

  # Retrieve merged headers prior to launching the pipeline. This is needed
  # since the BigQuery schema cannot yet be dynamically created based on input.
  # See https://issues.apache.org/jira/browse/BEAM-2801.
  header_fields = vcf_header_parser.get_vcf_headers(
      known_args.representative_header_file)
  counter_factory = metrics_util.CounterFactory()
  processed_variant_factory = processed_variant.ProcessedVariantFactory(
      header_fields,
      known_args.split_alternate_allele_info_fields,
      known_args.allow_malformed_records,
      known_args.annotation_fields,
      known_args.use_allele_num,
      known_args.minimal_vep_alt_matching,
      known_args.infer_annotation_types,
      counter_factory)

  schema = schema_converter.generate_schema_from_header_fields(
      header_fields, processed_variant_factory, variant_merger,
      known_args.use_1_based_coordinate,
      known_args.include_call_name,
      known_args.move_hom_ref_calls)

  sharding = variant_sharding.VariantSharding(known_args.sharding_config_path)
  if sharding.should_keep_shard(sharding.get_residual_index()):
    num_shards = sharding.get_num_shards()
  else:
    num_shards = sharding.get_num_shards() - 1

  if known_args.update_schema_on_append:
    for i in range(num_shards):
      table_suffix = sharding.get_output_table_suffix(i)
      table_name = bigquery_util.compose_table_name(known_args.output_table,
                                                    table_suffix)
      bigquery_util.update_bigquery_schema_on_append(schema.fields, table_name)

  pipeline = beam.Pipeline(options=beam_pipeline_options)
  variants = _read_variants(
      all_patterns, pipeline, known_args, pipeline_mode,
      use_1_based_coordinate=known_args.use_1_based_coordinate,
      move_hom_ref_calls=known_args.move_hom_ref_calls)
  if known_args.allow_malformed_records:
    variants |= 'DropMalformedRecords' >> filter_variants.FilterVariants()
  sharded_variants = variants | 'ShardVariants' >> beam.Partition(
      shard_variants.ShardVariants(sharding), sharding.get_num_shards())
  variants = []
  for i in range(num_shards):
    suffix = sharding.get_output_table_suffix(i)
    # Convert tuples to list
    variants.append(sharded_variants[i])
    if variant_merger:
      variants[i] |= ('MergeVariants' + suffix >>
                      merge_variants.MergeVariants(variant_merger))
    variants[i] |= (
        'ProcessVariants' + suffix >>
        beam.Map(processed_variant_factory.create_processed_variant). \
        with_output_types(processed_variant.ProcessedVariant))
    _ = (
        variants[i] | 'VariantToAvro' + suffix >>
        variant_to_avro.VariantToAvroFiles(
            avro_root_path + suffix,
            schema,
            allow_incompatible_records=known_args.allow_incompatible_records,
            omit_empty_sample_calls=known_args.omit_empty_sample_calls,
            null_numeric_value_replacement=(
                known_args.null_numeric_value_replacement),
            include_call_name=known_args.include_call_name,
            move_hom_ref_calls=known_args.move_hom_ref_calls)
    )
  result = pipeline.run()
  try:
    state = result.wait_until_finish()
    if state != beam.runners.runner.PipelineState.DONE:
      logging.error('Dataflow pipeline terminated in an unexpected state: %s',
                    state)
      raise AssertionError(
          'Dataflow pipeline terminated in {} state'.format(state))
  except Exception as e:
    logging.error('Dataflow pipeline failed.')
    raise e
  else:
    logging.info('Dataflow pipeline finished successfully.')
    metrics_util.log_all_counters(result)

  # After pipeline is done, create output tables and load AVRO files into them.
  schema_file = _write_schema_to_temp_file(schema, avro_root_path)
  suffixes = []
  try:
    for i in range(num_shards):
      suffixes.append(sharding.get_output_table_suffix(i))
      partition_range_end = sharding.get_output_table_partition_range_end(i)
      if not known_args.append:
        table_name = bigquery_util.compose_table_name(known_args.output_table,
                                                      suffixes[i])
        partitioning.create_bq_table(
            table_name, schema_file,
            bigquery_util.ColumnKeyConstants.START_POSITION,
            partition_range_end)
        _record_newly_created_table(table_name)
        logging.info('Integer range partitioned table %s was created.',
                     table_name)
    if not known_args.append:
      _record_newly_created_table(
          sample_info_table_schema_generator.create_sample_info_table(
              known_args.output_table))

    suffixes.append(sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)

    # If creating sample optimized tables and running in append mode, create
    # temp tables first, to be able to copy just new records to sample
    # optimized tables.
    if (known_args.append and known_args.sample_lookup_optimized_output_table):
      tmp_prefix = '_tmp_'
      temp_table_base_name = bigquery_util.compose_temp_table_base(
          known_args.output_table, tmp_prefix)

      temp_suffixes = []
      for i in range(num_shards):
        temp_suffixes.append(sharding.get_output_table_suffix(i))
        temp_partition_range_end = sharding.get_output_table_partition_range_end(
            i)
        temp_table_name = bigquery_util.compose_table_name(
            temp_table_base_name, temp_suffixes[i])
        partitioning.create_bq_table(
            temp_table_name, schema_file,
            bigquery_util.ColumnKeyConstants.START_POSITION,
            temp_partition_range_end)
        _record_newly_created_table(temp_table_name)
        logging.info('Integer range partitioned table %s was created.',
                     temp_table_name)
        _record_new_temp_table(temp_table_name)
      temp_sample_table_id = sample_info_table_schema_generator.create_sample_info_table(
          temp_table_base_name)
      _record_newly_created_table(temp_sample_table_id)
      _record_new_temp_table(temp_sample_table_id)
      temp_suffixes.append(
          sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)
      temp_load_avro = avro_util.LoadAvro(avro_root_path, temp_table_base_name,
                                          temp_suffixes, False)
      temp_not_empty_variant_suffixes = temp_load_avro.start_loading()

      # Copy tables
      for temp_t in _new_temp_tables:
        try:
          output_table = bigquery_util.get_non_temp_table_name(
              temp_t, tmp_prefix)
          bigquery_util.copy_table(temp_t, output_table)
        except Exception as e:
          logging.error(
              'Something unexpected during the copy of the temp '
              'table: %s to the target table %s: %s', temp_t, output_table,
              str(e))

      # Remove sample_info table from both lists to avoid duplicating it when
      # --sample_lookup_optimized_output_table flag is set
      temp_suffixes.remove(
          sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)
      if sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX in temp_not_empty_variant_suffixes:
        temp_not_empty_variant_suffixes.remove(
            sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)

      # Copy to sample optimized tables
      temp_flatten_call_column = partitioning.FlattenCallColumn(
          temp_table_base_name, temp_not_empty_variant_suffixes,
          known_args.append)
      try:
        temp_flatten_schema_file = tempfile.mkstemp(
            suffix=_BQ_SCHEMA_FILE_SUFFIX)[1]
        if not temp_flatten_call_column.get_flatten_table_schema(
            temp_flatten_schema_file):
          raise ValueError('Failed to extract schema of flatten table')
        # Copy to flatten sample lookup tables from the variant lookup tables.
        temp_flatten_call_column.copy_to_flatten_table(
            known_args.sample_lookup_optimized_output_table)
        logging.info('All sample lookup optimized tables are fully loaded.')
      except Exception as e:
        logging.error(
            'Something unexpected happened during the loading rows to '
            'sample optimized table stage. Since this copy failed, the '
            'temporary tables were not deleted. To avoid extra storage '
            'charges, delete the temporary tables in your dataset that '
            'will begin with %s. Error: %s', tmp_prefix, str(e))
        raise e
      else:
        for temp_t in _new_temp_tables:
          if bigquery_util.delete_table(temp_t) != 0:
            logging.error('Deletion of temporary table "%s" has failed.',
                          temp_t)

    else:
      load_avro = avro_util.LoadAvro(avro_root_path, known_args.output_table,
                                     suffixes, False)
      not_empty_variant_suffixes = load_avro.start_loading()
      logging.info('Following tables were loaded with at least 1 row:')
      for suffix in not_empty_variant_suffixes:
        logging.info(
            bigquery_util.compose_table_name(known_args.output_table, suffix))
      # Remove sample_info table from both lists to avoid duplicating it when
      # --sample_lookup_optimized_output_table flag is set
      suffixes.remove(
          sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)
      if sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX in\
          not_empty_variant_suffixes:
        not_empty_variant_suffixes.remove(
            sample_info_table_schema_generator.SAMPLE_INFO_TABLE_SUFFIX)

      if known_args.sample_lookup_optimized_output_table:
        flatten_call_column = partitioning.FlattenCallColumn(
            known_args.output_table, not_empty_variant_suffixes,
            known_args.append)
        try:
          flatten_schema_file = tempfile.mkstemp(
              suffix=_BQ_SCHEMA_FILE_SUFFIX)[1]
          if not flatten_call_column.get_flatten_table_schema(
              flatten_schema_file):
            raise ValueError('Failed to extract schema of flatten table')

          # Create all sample optimized tables including those that will be empty.
          for suffix in suffixes:
            output_table_id = bigquery_util.compose_table_name(
                known_args.sample_lookup_optimized_output_table, suffix)
            partitioning.create_bq_table(
                output_table_id, flatten_schema_file,
                bigquery_util.ColumnKeyConstants.CALLS_SAMPLE_ID,
                partitioning.MAX_RANGE_END)
            _record_newly_created_table(output_table_id)
            logging.info('Sample lookup optimized table %s was created.',
                         output_table_id)
          # Copy to flatten sample lookup tables from the variant lookup tables.
          flatten_call_column.copy_to_flatten_table(
              known_args.sample_lookup_optimized_output_table)
          logging.info('All sample lookup optimized tables are fully loaded.')
        except Exception as e:
          logging.error(
              'Something unexpected happened during the loading rows to '
              'sample optimized table stage: %s', str(e))
          raise e

  except Exception as e:
    logging.error('Something unexpected happened during the loading of AVRO '
                  'files to BigQuery: %s', str(e))
    logging.info('Since the write to BigQuery stage failed, we did not delete '
                 'AVRO files in your GCS bucket. You can manually import them '
                 'to BigQuery. To avoid extra storage charges, delete them if '
                 'you do not need them, AVRO files are located at: %s',
                 avro_root_path)
    raise e
  else:
    logging.warning('All AVRO files were successfully loaded to BigQuery.')
    if known_args.keep_intermediate_avro_files:
      logging.info('Since "--keep_intermediate_avro_files" flag is set, the '
                   'AVRO files are kept and stored at: %s', avro_root_path)
    else:
      if bigquery_util.delete_gcs_files(avro_root_path) != 0:
        logging.error('Deletion of intermediate AVRO files located at "%s" has '
                      'failed.', avro_root_path)

if __name__ == '__main__':
  logging.getLogger().setLevel(logging.INFO)
  try:
    run()
  except Exception as e:
    if _newly_created_tables:
      logging.info('Trying to delete all newly created tables.')
      bigquery_util.rollback_newly_created_tables(_newly_created_tables)
    else:
      logging.warning(
          'Since tables were appended, added rows cannot be reverted. You can '
          'utilize BigQuery snapshot decorators to recover your table up to 7 '
          'days ago. For more information please refer to: '
          'https://cloud.google.com/bigquery/table-decorators '
          'Here is the list of tables that you need to manually rollback:')
    raise e


text:
<?php

namespace ccxt;

class AuthenticationError extends ExchangeError {

}


text:
/*
 * LdapObjectDcObject.h
 *
 *  Created on: 04.01.2017
 *      Author: felix
 */

#ifndef SRC_LDAPOBJECTDCOBJECT_H_
#define SRC_LDAPOBJECTDCOBJECT_H_

#include <common/GenericLdapObject.h>

namespace Flix {

class LdapObjectDcObject: public GenericLdapObject {
public:
    LdapObjectDcObject();
    virtual ~LdapObjectDcObject();

    virtual bool isValid(void) const;
};

} /* namespace Flix */

#endif /* SRC_LDAPOBJECTDCOBJECT_H_ */


text:
using System;
using System.ComponentModel;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Net;
using XenAdmin.Core;
using XenAPI;
using System.Threading;
using XenAdmin.Network;

namespace XenAdmin.Actions
{
    public class ImportVmAction : AsyncAction
    {
        private static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
        private const int HTTP_PUT_TIMEOUT = 30*60*1000; //30 minutes

        private string m_filename;
        private readonly Host m_affinity;
        public const string IMPORT_TASK = "import_task";

		private bool m_startAutomatically;
    	private List<Proxy_VIF> m_proxyVIFs;

    	private object monitor = new object();
		private bool m_wizardDone;

    	/// <summary>
        /// These calls are always required to import a VM, as opposed to the set which are only called if needed set in the constructor of this action.
        /// </summary>
        public readonly static RbacMethodList ConstantRBACRequirements = new RbacMethodList(
            "vm.set_name_label",
            "network.destroy",
            "vif.create",
            "vif.destroy",
            "http/put_import"
        );

		public ImportVmAction(IXenConnection connection, Host affinity, string filename, SR sr)
			: base(connection, string.Format(Messages.IMPORTVM_TITLE, filename, Helpers.GetName(connection)), Messages.IMPORTVM_PREP)
		{
			Pool = Helpers.GetPoolOfOne(connection);
			m_affinity = affinity;
			Host = affinity ?? connection.Resolve(Pool.master);
			SR = sr;
			VM = null;
			m_filename = filename;

			#region RBAC Dependencies

			ApiMethodsToRoleCheck.AddRange(ConstantRBACRequirements);

			if (affinity != null)
				ApiMethodsToRoleCheck.Add("vm.set_affinity");

			//??
			//if (startAutomatically)
			//	ApiMethodsToRoleCheck.Add("vm.start");

			ApiMethodsToRoleCheck.AddRange(Role.CommonTaskApiList);
			ApiMethodsToRoleCheck.AddRange(Role.CommonSessionApiList);

			#endregion
		}

    	private string GetVmRef(string result)
        {
            if (string.IsNullOrEmpty(result))
                return null;

            string head = "<value><array><data><value>";
            string tail = "</value></data></array></value>";

            if (!result.StartsWith(head) || !result.EndsWith(tail))
                return null;

            int start = head.Length;
            int length = result.IndexOf(tail) - start;

            return result.Substring(start, length);
        }

        protected override void Run()
        {
            SafeToExit = false;
        	bool isTemplate;

        	try
            {
                string vmRef;

				if (m_filename.EndsWith("ova.xml"))//importing version 1 from of VM
				{
					m_filename = m_filename.Replace("ova.xml", "");
					vmRef = GetVmRef(applyVersionOneFiles());
				}
				else//importing current format of VM
					vmRef = GetVmRef(applyFile());

            	if (Cancelling)
                    throw new CancelledException();

                // Now lets try and set the affinity and start the VM

                if (string.IsNullOrEmpty(vmRef))
                    return;

                while (!Cancelling && (VM = Connection.Resolve(new XenRef<VM>(vmRef))) == null)
                    Thread.Sleep(100);

                if (Cancelling)
                    throw new CancelledException();

                isTemplate = VM.get_is_a_template(Session, vmRef);
                Description = isTemplate ? Messages.IMPORT_TEMPLATE_UPDATING_TEMPLATE : Messages.IMPORTVM_UPDATING_VM;
                VM.set_name_label(Session, vmRef, DefaultVMName(VM.get_name_label(Session, vmRef)));

				if (!isTemplate && m_affinity != null)
					VM.set_affinity(Session, vmRef, m_affinity.opaque_ref);

                // Wait here for the wizard to finish
                Description = isTemplate ? Messages.IMPORT_TEMPLATE_WAITING_FOR_WIZARD : Messages.IMPORTVM_WAITING_FOR_WIZARD;
				lock (monitor)
				{
					while (!(m_wizardDone || Cancelling))
						Monitor.Wait(monitor);
				}

                if (Cancelling)
                    throw new CancelledException();

                if (m_proxyVIFs != null)
                {
                    Description = isTemplate ? Messages.IMPORT_TEMPLATE_UPDATING_NETWORKS : Messages.IMPORTVM_UPDATING_NETWORKS;

                    // We need to destroy all vifs and recreate them.

                    List<XenRef<VIF>> vifs = VM.get_VIFs(Session, vmRef);
                    List<XenAPI.Network> networks = new List<XenAPI.Network>();

                    foreach (XenRef<VIF> vif in vifs)
                    {
                        // Save the network as we may have to delete it later
                        XenAPI.Network network = Connection.Resolve(VIF.get_network(Session, vif));
                        if (network != null)
                            networks.Add(network);

                        VIF.destroy(Session, vif);
                    }

                    foreach (Proxy_VIF proxyVIF in m_proxyVIFs)
                    {
                        VIF vif = new VIF(proxyVIF) {VM = new XenRef<VM>(vmRef)};
                        VIF.create(Session, vif);
                    }

                    // now delete any Networks associated with this task if they have no VIFs

                    foreach (XenAPI.Network network in networks)
                    {
                        if (!network.other_config.ContainsKey(IMPORT_TASK))
                            continue;

                        if (network.other_config[IMPORT_TASK] != RelatedTask.opaque_ref)
                            continue;

                        try
                        {
                            if (XenAPI.Network.get_VIFs(Session, network.opaque_ref).Count > 0)
                                continue;

                            if (XenAPI.Network.get_PIFs(Session, network.opaque_ref).Count > 0)
                                continue;

                            XenAPI.Network.destroy(Session, network.opaque_ref);
                        }
                        catch (Exception e)
                        {
                            log.ErrorFormat("Exception while deleting network {0}. Squashing.", network.Name);
                            log.Error(e, e);
                        }
                    }
                }

                if (!VM.get_is_a_template(Session, vmRef))
                {
                    if (m_startAutomatically)
                    {
                        Description = Messages.IMPORTVM_STARTING;
                        VM.start(Session, vmRef, false, false);
                    }
                }
            }
            catch (CancelledException)
            {
                Description = Messages.CANCELLED_BY_USER;
                throw;
            }

            Description = isTemplate ? Messages.IMPORT_TEMPLATE_IMPORTCOMPLETE : Messages.IMPORTVM_IMPORTCOMPLETE;
        }

        private int VMsWithName(string name)
        {
            int i = 0;

            foreach (VM v in Connection.Cache.VMs)
                if (v.name_label == name)
                    i++;

            return i;
        }

        private string DefaultVMName(string vmName)
        {
            string name = vmName;
            int i = 0;

            if (VMsWithName(vmName) > 1)
                while (VMsWithName(name) > 0)
                {
                    i++;
                    name = string.Format(Messages.NEWVM_DEFAULTNAME, vmName, i);
                }

            return name;
        }

        private List<string> TaskErrorInfo()
        {
            return new List<string>(Task.get_error_info(Session, RelatedTask));
        }

        private long getSize(DirectoryInfo dir, long count)
        {
            long s = count;
            foreach (FileInfo f in dir.GetFiles("*.gz"))
                s += f.Length;

            foreach (DirectoryInfo d in dir.GetDirectories())
                s += getSize(d, s);

            return s;
        }

        private string applyVersionOneFiles()
        {
            RelatedTask = Task.create(Session, "importTask", Messages.IMPORTING);

            try
            {
				long totalSize = getSize(new DirectoryInfo(m_filename), 0);
                long bytesWritten = 0;

                if (totalSize == 0)
                {
                    // We didn't find any .gz files, just bail out here
                    throw new Exception(Messages.IMPORT_INCOMPLETE_FILES);
                }

            	CommandLib.Config config = new CommandLib.Config
            	                           	{
            	                           		hostname = Connection.Hostname,
            	                           		username = Connection.Username,
            	                           		password = Connection.Password
            	                           	};

                CommandLib.thinCLIProtocol tCLIprotocol = null;
                int exitCode = 0;
            	tCLIprotocol = new CommandLib.thinCLIProtocol(delegate(string s) { throw new Exception(s); },
            	                                              delegate { throw new Exception(Messages.EXPORTVM_NOT_HAPPEN); },
            	                                              delegate(string s, CommandLib.thinCLIProtocol t) { log.Debug(s); },
            	                                              delegate(string s) { log.Debug(s); },
            	                                              delegate(string s) { log.Debug(s); },
            	                                              delegate { throw new Exception(Messages.EXPORTVM_NOT_HAPPEN); },
            	                                              delegate(int i)
            	                                              	{
            	                                              		exitCode = i;
            	                                              		tCLIprotocol.dropOut = true;
            	                                              	},
            	                                              delegate(int i)
            	                                              	{
            	                                              		bytesWritten += i;
            	                                              		PercentComplete = (int)(100.0*bytesWritten/totalSize);
            	                                              	},
            	                                              config);

                string body = string.Format("vm-import\nsr-uuid={0}\nfilename={1}\ntask_id={2}\n",
											SR.uuid, m_filename, RelatedTask.opaque_ref);
				log.DebugFormat("Importing Geneva-style XVA from {0} to SR {1} using {2}", m_filename, SR.Name, body);
                CommandLib.Messages.performCommand(body, tCLIprotocol);

                // Check the task status -- Geneva-style XVAs don't raise an error, so we need to check manually.
                List<string> excep = TaskErrorInfo();
                if (excep.Count > 0)
                    throw new Failure(excep);

                // CA-33665: We found a situation before were the task handling had been messed up, we should check the exit code as a failsafe
				if (exitCode != 0)
					throw new Failure(new[] {Messages.IMPORT_GENERIC_FAIL});

                return Task.get_result(Session, RelatedTask);
            }
            catch (Exception exn)
            {
                List<string> excep = TaskErrorInfo();
                if (excep.Count > 0)
                    throw new Failure(excep);
                else
                    throw exn;
            }
            finally
            {
                Task.destroy(Session, RelatedTask);
            }
        }

        private string applyFile()
        {
            log.DebugFormat("Importing Rio-style XVA from {0} to SR {1}", m_filename, SR.Name);

            Host host = SR.GetStorageHost();

            string hostURL;
            if (host == null)
            {
                Uri uri = new Uri(Session.Url);
                hostURL = uri.Host;
            }
            else
            {
                log.DebugFormat("SR is not shared -- redirecting to {0}", host.address);
                hostURL = host.address;
            }

            log.DebugFormat("Using {0} for import", hostURL);

            return HTTPHelper.Put(this, HTTP_PUT_TIMEOUT, m_filename, hostURL,
                                  (HTTP_actions.put_ssbbs)HTTP_actions.put_import,
                                  Session.uuid, false, false, SR.opaque_ref);
        }

		public override void RecomputeCanCancel()
		{
		    CanCancel = m_wizardDone;
		}

        protected override void CancelRelatedTask()
        {
            log.InfoFormat("VM/Template import from '{0}' cancelled", m_filename);

			lock (monitor)
			{
			    Monitor.PulseAll(monitor);
			}

            base.CancelRelatedTask();
        }

		public void EndWizard(bool start, List<Proxy_VIF> vifs)
		{
			m_startAutomatically = start;
			m_proxyVIFs = vifs;

			lock (monitor)
			{
				m_wizardDone = true;
				Monitor.PulseAll(monitor);
			}
		}
    }
}


text:
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent" >

    <GridView
        android:id="@+id/gridView1"
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"
        android:layout_marginBottom="20dp"
        android:numColumns="3" >
    </GridView>

</RelativeLayout>


text:
var connect = require('connect');
var serveStatic = require('serve-static');
var port = process.argv[2] || 8888;

connect().use(serveStatic(__dirname)).listen(port);

console.log("Static file server running at\n  => http://localhost:" + port + "/\nCTRL + C to shutdown");