text:
\section{Introduction.}
In this paper we revisit a classic question: how does auction design affect revenues and bidder behavior? We shed a new light on this question by analyzing auctions where bidders use simple artificial intelligence algorithms to determine their bids (rather than using the Nash equilibrium paradigm).
Our main result is that when bidders use these algorithms, auction format and other design choices can have a first-order effect on revenues and bidder payoffs.

In particular, we present two findings. First, we observe that revenues can be significantly lower in first-price auctions than in second-price auctions. We show this in a very simple setup: with two bidders and constant symmetric values. The algorithms we consider are simple Q-learning algorithms that keep track of the history of past auctions in a very reduced form and are not explicitly designed to discover dynamic reward/punishment strategies.
Second, we show that a simple practical auction design choice --- revealing bidders the winning bid after the auction --- can make the first-price auctions much more competitive.\footnote{When Google switched in 2019 the Google Ad Manager auctions for display advertising from the second-price to the first-price format, they simultaneously started providing such information to all buyers bidding in that auction: "Buyers will receive the minimum bid price to win after the auction closes" - see \url{https://blog.google/products/admanager/rolling-out-first-price-auctions-google-ad-manager-partners/} .}  In fact, providing such information can bring bids and revenues to the competitive level predicted by a one-shot Nash equilibrium (and to the level we observe in second-price auctions).

We are interested in studying how auction design affects play by AI algorithms for two reasons. First, we think that our analysis is quite applicable to online advertising auctions, where bidders for some types of impressions/clicks compete in many thousands of auctions a day.\footnote{The number varies greatly with the website and the target audience. To get a sense of the volumes, the New York Times website is estimated to be visited 339M times in November 2021 and visitors viewed on average 2.28 pages per visit - see \url{https://www.similarweb.com/website/nytimes.com}. With even only one ad per page, that is over 25M ad impressions sold a day. Estimates of the number of daily searches (and hence opportunities to run keyword auctions) on Google's search engine are in billions a day. Clearly, any given advertiser is interested in only a fraction of those advertising opportunities, yet we still expect that many advertisers participate in thousands of auctions a day.}
Online ad auctions usually happen in a fraction of a second. It is common for bidders to rely on bidding tools either provided by the auctioneers, developed in-house or by third parties.\footnote{There are many such examples: Smart Bidding at Google (\url{https://support.google.com/google-ads/answer/7065882?hl=en}) is described as "a subset of automated bid strategies that use machine learning to optimize for conversions or conversion value in each and every auction." An Example of a third-party algorithm is scibids.com that states "We build customizable AI that dramatically improves Paid Media ROI."}. As such algorithms grow in popularity, it is a natural question to ask how play in such auctions would evolve when multiple competitors use similar tools, each optimizing on behalf of its owner.

Second, a collection of recent papers on algorithmic pricing expressed concern that algorithms used for pricing could facilitate (tacit) collusion (see the related literature discussion). The general theme is that sophisticated algorithms can learn to play dynamic strategies, offering rewards to their competitors for tacit collusion and punishing them for competitive behavior --- playing a low-revenue equilibrium of an infinitely repeated game instead of the high-revenue equilibrium of the static game. Such behavior by algorithms can be facilitated by their ability to continuously monitor each other and react (with punishments) at much higher speeds than humans can. Our theory of repeated games suggests that improved monitoring and speed of reaction make tacit-collusive equilibria more stable. We are interested in understanding to what extent the repeated games intuitions apply to simple AI algorithms and whether auction design choices can affect those forces.

The simple AI algorithm we use in this paper is Q-learning, one of the most popular algorithms for agnostic learning. We consider the simplest Q-learning algorithm that works roughly as follows: for a finite set of available bids, a bidder keeps track of a Q-vector, that is its current estimate of the value of taking any action (the Q-vector has one entry for each potential bid). The Q-vector tries to estimate both current and future payoff consequences of choosing any particular bid today. The algorithm chooses with probability $1-\epsilon$ in any given period a bid that corresponds to the highest expected entry in Q. With probability $\epsilon$ it takes any other action. As the player observes its current payoff, he updates its Q estimate, putting a higher weight on recent data. In our main specification, epsilon decays over time, capturing the tradeoff between exploration and exploitation.\footnote{We also show some results in case epsilon remains positive even in the long run and they are the same as our main finding. A more complex Q-learning algorithm would include additional states - for example, allowing the estimate of Q to depend on recent history, like who won the last auction. Such more sophisticated algorithms are of interest. We do not discuss them in this paper since they are much harder to analyze and because they are by design guiding bidders towards tacit collusion. One of the things we are interested in this paper is understanding whether tacit collusion is a concern even with simple AI algorithms.}

The Q-learning algorithm has been shown to have great success in finding optimal strategies in single-person decision making problems — see for example \citet{Watkins1992}. In our setup, the algorithm performs very well at finding the best response strategy to any fixed strategy of its opponents. Additionally, Q-learning algorithms are the basic building blocks of more sophisticated AI algorithms, therefore we think understanding the dynamics of bidding by Q-learning agents is going to shed light on likely consequences of auction design in more complex environments.

Our first main finding is shown in Figure \ref{fig:base}, representing the distribution of long-run behavior for our bidding agents over 1,000 experiments that each involve 1,000,000 auctions. The results are stark: bidders in the repeated second-price auctions converge to bidding according to the static Nash equilibrium prediction and the revenues to the auctioneer are high. In contrast, in the first-price auctions bidders converge over time to much lower bids (the average bid is 0.24) --- this seems to be reminiscent of tacit collusion.\footnote{ These results are robust to not letting the experimentation rate $\epsilon$ decay to zero --- see Figure \ref{fig:exp}. }

Our second main finding is shown in Figure \ref{fig:synchronous}: when in the first-price auction bidders observe not only whether they won, but also what was the lowest bid to guarantee winning, they can update the Q vector not only for the current bid, but for all counterfactual bids (so-called synchronous updating). It turns out that with this additional information, even in the first-price auction bidders converge over time to highly competitive bids.

To understand the economic forces behind the first result, we notice that there are multiple differences between the first-price and second-price auction formats that can contribute to the difference in outcomes, despite the theory of repeated games suggesting that tacit collusion should be approximately equally easy/hard in those two games. In particular, we point out that:
\begin{enumerate}
    \item The second-price one-shot auction has a dominant bidding strategy while the first-price auction does not, and the algorithm is better at finding dominant strategies.
    \item When bidders try to coordinate on a low bid, then in the second-price auction all higher bids are profitable (in the short run) while in the first-price auction only bids close to the current bid are profitable. For example, if bidders coordinate on bids (0.2, 0.2), then any bid between 0.2 and 1 is profitable in the SPA while only bids  between 0.2 and 0.6 are profitable in the FPA. Since our algorithms discover profitable deviations via random experimentation, it may be harder for them to find those in FPA, some tries to a higher bid (for example, 0.8) can result in lower profits than current tacitly collusive bid, and hence bidders may stop experimentation.
    \item Because there are so many more potentially profitable deviations, it is possible that it is harder for the algorithms to coordinate/reach the tacit-collusive outcome in the second-price auctions than in first-price auctions, so the difference comes from the algorithms' initialization. However, with an initialization on low bids, the long-term conduct could be the same
    \item The two auctions are different in terms of how the algorithms behave when they mis-coordinate at different bids. Namely, when bidder one outbids bidder two, in both formats bidder two will learn that their current strategy is not profitable and will start exploring other bids. The difference is in how bidder one (the higher bidder) behaves in these two formats. In the second-price auction, the payoffs of that bidder do not depend on his current bid, hence there is no first-order force to push him to higher or lower bids. On the other hand, in the first-price auction, a winning bidder has incentives to win by as little as possible. So if the bidders start with bids (0.2, 0.2), the deviating algorithm is going to learn that against a constant opponent bid the optimal deviation is just by one bid increment (by 0.05 in our simulations). As a result, when the bidders submit equal bids, they on average hit lower bids in the first-price auction than in the second-price auction.
\end{enumerate}

We designed additional experiments/simulations to tell these possible explanations apart. It turns out that the main reason for the difference is the last one. This force is a fundamental difference between the first-price and second-price auctions. It also helps us understand why the tacit collusion seems to lead to an average revenue 0.24 per auction and not less. When bidders try to coordinate on very low bids, experimentation pushes them only up not down, and they are very unlikely to return to very low bids.

To understand the forces behind the second result, it is helpful to think of the incentives to re-coordinate on lower bids. Imagine that both bidders coordinate on a bid of $0.3$, until bidder one deviates to $0.4$. Bidder one will experience an immediate boost in his estimate of the new bid's value, while bidder two will surely realize that $0.3$ is no longer a good bid. After some experimentation, suppose the two coordinate on $0.4$. If the two bidders can only update the current bid, the bid $0.3$ remains biased: soon enough they will discover that coordinating on $0.4$ is worse than coordinating on $0.3$ and attempt to move back. If they update synchronously instead, using the counterfactual return from other bids, the estimate of the value of $0.3$ will drop dramatically: the return in hindsight is always zero. Synchronicity in some sense leads to shortsightedness: the counterfactual measure used does not take into account future re-coordinations.

We finish the paper by considering several extensions. We analyze the effect of reserve prices on bidder behavior and compare it with the introduction of a competitive fringe. We analyze the game with more competition - either with 3 strategic bidders or two bidders and a competitive fringe (as expected, competition increases revenues in the first-price auctions, but it does not eliminate the tacit-collusive outcomes).

\subsection{Related Literature.}

Algorithmic collusion has recently sparked some interest in the Economics community. The pioneering work of \citet{Calvano2020} examines collusion in a price-setting oligopoly, and suggests that algorithms  keeping track of past prices adopt collusive strategies typical of implicit cartels. They too study Q-learning as a workhorse Artificial Intelligence algorithm. \citet{Klein2021} also studies collusion and Q-learning in a pricing game, but in that setting price offers are alternated. In a similar pricing model, \citet{Asker2022} study the effect of algorithm design on collusion. Similar to their paper, we find that synchronous algorithms are less likely to converge on collusive outcomes. The strength of such finding in our setting is supported by Google's auction design choice, as described in the introduction.  \citet{Hansen2021} simulate a different algorithm from the bandit literature, and show how its misspecified implementation similarly results in collusion.

Empirical work on algorithmic pricing has also been flourishing. \citet{Brown2021} study the effect of high-frequency price adjustments on competition between online retailers. They find that higher frequency may lead to price dispersion and increase overall price levels.  \citet{Musolff2021} finds that online retailers' prices follow a Hedgeworth's cycles behavior through a price-resetting mechanic that decreases competition. In \citet{Assad2021} the authors document a rise in margins of retail gasoline sellers from adoption of automated pricing algorithms. Some recent work analyzes learning in auctions, but most of the literature is concerned with the auctioneer's side (\citet{Milgrom2019}) with the exception of \citet{Nedelec2019}: they analyze a strategic bidder's objective and approach learning through gradient descent.

The pioneering work on Q-learning by \citet{Watkins1989} and \citet{Watkins1992} revived a literature on reinforcement learning which has been explored briefly also in economics (\citet{Erev1998}, \citet{Erev1999}). The theory surrounding multi-agent learning has had great success in practical applications, but there is no consensus among computer scientists on a leading paradigm. For a recent overview, we refer the reader to \citet{Zhang2021}. The theoretical studies on multi-agent reinforcement learning have led to connections with the evolutionary game theory literature (see \citet{Bloembergen2015}), mostly for specific algorithms and rules. Recently, \citet{Banchio2022b} propose a continuous-time approximation for algorithmic systems which allows for characterization of equilibria. Our setting is too high-dimensional to apply such techniques, but we are able to test our hypotheses experimentally and build intuitions from their simple setup.

The literature on Learning in Games has analyzed systems of learners from various angles. The workhorse model of Fictitious Play by \citet{Brown1951} has been studied thoroughly and its properties are well understood (see \citet{Fudenberg1998} for a thorough review), but its practical adoption has been long shunned. The simple reinforcement learning models of \citet{Erev1998} and \citet{Borgers1997} produce interesting predictions with simple algorithms. However, most of the literature is concerned with learning as a foundation for Nash equilibrium and equilibrium concepts as a whole. In this work we are interested in the equilibrium behavior of learning systems instead, where the learning system is taken as given and equilibria are hard to characterize. As we show, in some auction formats the algorithms do not converge to the Nash equilibrium of the static game. This means that in a play where experimentation does not die out, they do not converge to any one action profile. Instead they end up in stochastic cycles, with long-term average bids substantially lower than in the Nash equilibrium of the static game.

The final literature our paper is related to is theoretical literature on collusion in auctions. \citet{Mcafee1992} discuss collusive schemes by strong and weak cartels, defined as those that use and do not use side payments. They discuss the properties of the bid rotation schemes and under what conditions bidders can benefit from them. \citet{Skrzypacz2004} study tacit collusion in repeated games where the bidders observe publicly only the identity of the winner (see also \citet{Aoyagi2003} for further analysis of bid rotation in repeated auctions). \citet{athey2001} study tacit collusion when players observe all bids. \citet{Marshall2009} show how the extensive form of auction formats may inhibit or facilitate tacit collusion. Our contribution is to show the interaction between auction design and level of competition while assuming strategies are chosen by the algorithms rather than they are dictated by rationality assumptions.

\section{The Model.}\label{sec:model}
Two bidders participate in a sequence of auctions.\footnote{Some of our simulations consider more than two bidders; we explain those extensions later.} In every period $t \in \{1,\dots,\infty\}$ an auctioneer runs an auction to allocate a single non-divisible object to one of the bidders. Both bidders value the object at $v_i = 1$ and the value is constant over time.

We consider a family of auction formats parameterized by $\alpha \in [1,2]$. In an $\alpha-$auction the highest bidder wins and pays a convex combination of the winning and the losing bid. The weight on the losing bid is $\alpha-1$, and the weight on the winning bid is $2-\alpha$. We focus on the two extreme cases: the first-price auction (FPA, $\alpha=1$) and the second-price auction (SPA, $\alpha=2$).

The payoff of the winner of period $t$ auction is $\pi_t = 1-p_t$ where $p_t$ is the price determined by the mechanism chosen by the auctioneer. The losing bidder gets a payoff of $0$. Bidders maximize the expected sum of discounted per-period payoffs with a discount factor $\gamma \in (0,1)$.

In the auction, bidders choose from a finite grid of prices. Each bidder has access to a set of equidistant bids $[b_1, \dots, b_m]$ where $b_i = \frac{i}{m+1}$. This assumption allows us to work with simpler learning algorithms. It is also representative of auctions typically allowing bidders to submit bids expressed in dollars and cents. Some online auctions restrict the bids even further.\footnote{For example, in Capterra.com auctions ''Bids start at \$2  per click and can be increased in \$ 0.25 increments.'' See https://blog.capterra.com/what-is-ppc/.} Note that we restrict attention to bids smaller than the bidder values.\footnote{We also assume that the highest bid is strictly less than value so that in the static Nash equilibrium, bidders play strict best responses. When the highest available bid is $1$, the payoffs are zero in the static Nash equilibrium, and bidders are indifferent between following the equilibrium strategy and any deviation to a lower bid. To ensure this indifference does not drive our results, we keep a small wedge between the highest bid and the value. An additional benefit of our assumptions about the grid is that the equilibrium of the second-price auction is unique (while it is not unique when bids can be arbitrary).}

\subsection{Nash Equilibria of the Auctions.}

This paper aims to study how auction design (for example, a choice between the first-price and second-price auction) affects the outcomes if the players use simple artificial intelligence algorithms to choose their bids.
To put our results in perspective, we first discuss Nash equilibria of the auctions from the repeated and static perspectives.

Given our very simple environment, for all $\alpha$, the one-shot game has a unique Nash equilibrium, with both players bidding $b_m$ (the largest bid bellow value). In equilibrium, bidders have strict incentives to follow the equilibrium strategy. In these static equilibria,  auctioneer revenues are independent of the format.

If the discount factor is sufficiently high, the repeated game has many other equilibria. The set of equilibria depends on the information provided to the bidders after every auction. Do they only observe whether they won and their price? Or do they observe both bids? Alternatively, do they observe something else?
The analysis of the equilibria of the repeated auctions is simpler when the bidders observe both bids after the auction. That makes it a game with perfect public monitoring. If bidders only observe their bids and whether they won, this becomes a game with imperfect public monitoring. It is often the choice of the auctioneer how much information to reveal to the players. We analyze the consequences of such a design choice in Section 4.\footnote{For example, in FPA, the auctioneer can choose to reveal or hide the losing bid from the winner and/or reveal the winning bid to the loser. As we mentioned in the Introduction, in a recent switch from SPA to FPA, Google decided to reveal both the winning bid to losers and the highest losing bid to the winner).}

To keep this discussion short, we review only some of the equilibria. In Appendix \ref{app:equilibria} we first discuss strongly symmetric equilibria when the bidders observe both bids. Then we consider Bid Rotation equilibria that require only that the bidders observe the identity of the winner\footnote{For an analysis of Perfect Public Equilibria of the repeated FPA and SPA with private values and bidders publicly observing only the identity of the winner see \citet{Skrzypacz2004}}. The takeaway from the repeated games literature is that, with perfect monitoring, strongly symmetric collusive equilibria are easier to sustain in the FPA than in the SPA, although the difference is minor.  Instead, if players only publicly observe the winner's identity, it is possible to sustain tacitly collusive outcomes even via simple public perfect equilibria, through bid rotation schemes. Moreover, the analysis suggests that these equilibria should be easier to sustain in SPA than in FPA.
The main takeaway from our results is that the propensity of simple algorithms to reach tacitly collusive outcomes depends on economic forces beyond the intuitions from the analysis of the repeated games.

\section{Q-Learning.}

First proposed by \citet{Watkins1989}, Q-learning algorithms are the main building blocks of the reinforcement learning paradigm. Actions found to be more profitable are more likely to be taken in the future: each result reinforces the agent's understanding of the environment.
Formally, in each period two agents choose a bid $b^i_t \in B = \{b_1,\dots,b_m\}$. The agents earns a stochastic reward $r_t$ distributed according to $F(r_t|b^i_t,b^{-i}_t)$.
We will work with a simple tabular environment with finite action set $B$ not just for simplicity but for interpretability: the model requires few hyperparameters with clear economic relevance, while the neural networks necessary to implement more complex Q-learning-based approaches do not easily lend themselves to interpretation. A convenient simplification is absence of states for the algorithm. While some papers design the learners to keep track of past actions, in the original Q-learning formulation states are Markovian parameters of the environment. In this sense, our environment is time-independent, and the algorithms do not need any additional information about play.\footnote{Q-learning is not misspecified here: rewards in each period depend only on the actions of the agent in that period. The ability to recall past actions serves as a monitoring technology, but does not have direct payoff implications. Contrast this with Markov games where the distribution of rewards conditions on the current state of the environment.} Additionally, some of the environments discussed in the introduction do not provide enough information to the algorithms for appropriate memory representation.

Each agent maximizes the discounted sum of rewards $\mathbb{E}\Big[\sum_t^\infty \gamma^t r_t\Big]$ where $\gamma <1$ is the discount factor. Instead of considering the value of Dynamic Programming, Q-learning estimates the action-value function:
\[
Q(a) =  \mathbb{E}[r|b^i,b^{-i}] + \gamma \mathbb{E}[\max_{b'} Q(b')]
\]
Notice that the optimal value is simply $V = \max_b Q(b)$. If the agent learns the Q-function, he can play the optimal strategy. The algorithms is simple: starting from an arbitrary initial action-value function, after choosing an action $a_t$  update the Q-function as follows:
\[
Q_{t+1}(b_t) =  (1-\alpha)Q_{t}(b_t) + \alpha\Big[r_t + \gamma \max_b Q_t(b)\Big]
\]

This particular form of learning is asynchronous: only the state-action pair visited in a particular period is updated, while the rest of the Q-function remains constant.
The hyperparameter $\alpha$ is called learning rate. Its task is to discipline the speed of learning, but also for how long past experience is retained in today's estimate of action-values. The updating procedure is a long-run average, and in this sense the parameter $\alpha$ is the counterpart of the discount factor: $\gamma$ determines the importance given to the future, while $\alpha$ specifies how quickly the algorithm forgets about the past.

\citet{Watkins1992} prove that Q-learning converges to the optimal policy in a Markov Decision Problem (MDP) for a single agent. However, no such guarantee exists for general multi-agent Q-learning. Difficulties arise from the loss of stationarity: each agent faces an unpredictable, ever-changing environment. The reward distribution depends on the opponents actions as well. One approach to multi-agent Q-learning considers opponents' past actions as part of the state, but essentially ignoring the endogeneity of transition and reward probabilities. While the Markov property is clearly not satisfied, various experiments in the literature find independent Q-learning to perform well in these settings, as is the case in our repeated auction. Additionally, opponent-aware algorithms would require more information about each opponent's design and behaviour, whereas the independent design approach retains the model-free philosophy of the reinforcement learning paradigm.

\paragraph{Experimentation.}
The Q-learning procedure specifies an update policy for every action taken, but it does not specify a choice of action directly.  In the algorithm proposed by Watkins, agents take actions uniformly at random: repetition guarantees that each sequence of actions will be taken sufficiently many times, and Q-learning will eventually visit every state and learn its value. This approach is limited in multiple ways, and particularly it fails to account for the tradeoff between exploration and exploitation: after an initial exploration period it would be reasonable to reap profits from actions that have been consistently outperforming.
We focus on a rule known as $\epsilon$-greedy: the agent takes the action that maximizes the Q-function with probability $1-\epsilon$, and takes an action uniformly at random with probability $\epsilon$. The exploration probability will take the form $\epsilon = \varepsilon e^{-\beta t}$, where $\beta$ further regulates the exploration-exploitation tradeoff as time progresses.

Another popular exploration paradigm, optimistic Q-learning, has shown some degree of success. Optimistic Q-learners are purely greedy: they always take the estimate of the optimal action at that point in time. However, the Q-function is initialized unusually high: for every state-action pair, the value of $Q$ is larger than the maximum payoff it could ever be achieved. The purpose of such an initialization is to ensure that experimentation will be pervasive: the algorithm won't stop experimenting until all of the $Q(a)$ values will have sufficiently decreased.\footnote{\citet{EvenDar2002} prove that optimistic tabular Q-learning converges to the correct optimal policy in MDPs.} In our multi-agent setting, the advantage offered by optimism is a phase of intense experimentation at the beginning, which improves convergence.

\section{Results.}
In most of our simulations, we simulate $2$ independent Q-learning algorithms bidding in the auctions. The grid of allowable bids includes 19 bid levels from $0.05$ to $0.95$. Unless otherwise stated, each experiment is repeated $1000$ times, and we terminate each run after 1 million periods. The algorithms have converged if the strategy does not change for the last 1,000 iterations, that is, if for each player $\argmax Q_i(a)$ stays constant.  We discard simulations that do not converge\footnote{Note that in the baseline specification, 1 million periods is enough to obtain convergence of nearly all experiments. However, to ensure robustness, we also replicate the experiments with 10 and 100 million periods, with nearly identical results.}. In the baseline specification, we adopt an $\epsilon$-greedy exploration policy with $\varepsilon = 0.025$, $\beta = 0.0002$, $\gamma = 0.99$, and $\alpha = 0.05$, with an optimistic initialization.

In our first set of results We compare bidding outcomes under the first-price auction (FPA) and second-price auction (SPA) formats. The outcomes of these experiments are reported in Figure \ref{fig:base}.

\begin{result}\label{Result1}
\item Our algorithms converge to the static Nash equilibrium in the second-price auction.
\item They converge to much lower bids in the first-price auction.
\item There is large dispersion of outcomes at convergence in FPA, and no dispersion in SPA.
\item The average (across simulations, at convergence) revenue per auction in SPA is $0.95$ and in FPA it is $0.24$.
\end{result}

\begin{figure}
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{base_FPA.png}
  \caption{FPA bids}
  \label{fig:baseFPA}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{base_SPA.png}
  \caption{SPA bids}
  \label{fig:baseSPA}
\end{subfigure}
\caption{The heatmaps above show the frequencies of each pair of strategies at convergence. The bids of the two algorithms are ordered along the $x$- and $y$-axis. }
\label{fig:base}
\end{figure}

One thing to notice in Figure \ref{fig:base} is that the algorithms do not converge on pairs of bids outside of the diagonal. This is intuitive since if they did, the losing algorithm would learn that their strategy gives payoff $0$ and would over time find out that deviating to bid $0.95$ would be profitable (a contradiction).

The difference between the outcomes for the two formats is rather striking: the two bidders converge to the static equilibrium in the case of SPA. But they ``collude'' on low bids in FPA. In particular, the average revenue of the auctioneer when using SPA is $0.95$ (the highest possible bid), while the average revenue under FPA is $0.24$.

Note that the equilibrium theory of repeated games that we discussed earlier is not good at predicting this outcome. As we explained, tacit collusion based on strongly symmetric equilibria is sustainable under both formats for similar assumptions, and tacit collusion based on asymmetric equilibria is much easier to sustain in the second-price auctions. A different view could be that since our algorithms do not explicitly keep track of history (they are not designed to learn conditional strategies like the bid rotation scheme), Nash equilibrium theory would predict that collusion should not be possible in either of the formats.

One may worry that the decay of experimentation drives our results. Our algorithms may be getting stuck at low bids in FPA because they stop deviating by the time they learn that there is a profitable deviation. To check that our Result \ref{Result1} is robust to other processes for experimentation, we also ran the auctions for $100$ million iterations, while keeping the experimentation parameter constant at $\varepsilon = 0.001$. The results are presented in Figure \ref{fig:exp} that counts the number of times out of the 100 million rounds the different pairs of bids have been played. Since these algorithms never stop experimenting, they continue to visit all pairs of bids. Yet, consistent with our findings in Result \ref{Result1}, a clear pattern appears: the algorithms spend most of the time in SPA at bids $(0.95,0.95)$ and much lower bids in FPA.

Interestingly, in FPA they do not spend all the time at one pair of low bids but move across them. This is consistent with the algorithms being good at finding best responses - if an opponent converged to a constant bid (in most periods), an algorithm in FPA would learn to bid just one bid increment more. So, the only candidate for convergence to constant bids is the static Nash equilibrium. Instead, the algorithms end up in a cycle, moving between several pairs of low bids.

Our next step is to formulate multiple hypothesis for the main economic forces that cause the differences and then design additional simulations to test them. We then try to also provide economic intuition for why in the first-price auction despite the algorithms seemingly learning to tacitly collude, they converge to bids far away from perfect collusion at $b_i=0.05.$

\subsection{Hypotheses.}
We now formulate and analyze a few possible explanations for the observed play by algorithms in the two formats. Our methodology is to design experiments/simulations for each hypothesis to test if it seems to be one of the key forces responsible for those results.

\paragraph{Dominant strategy.} One difference between FPA and SPA pointed out in the literature is the ``simplicity'' of the latter. The equilibrium strategy in the SPA is the unique weakly dominant strategy hence does not require conjectures about the play of the opponent. This is not true in the FPA - there the best response depends on the conjecture about the distribution of bids of the opponent.
Moreover, from the learning literature we know that strategic simplicity (for example, the game being dominance-solvable) is often enough to guarantee convergence on Nash equilibria.

To see whether dominance per se affects the results, we run the following experiment. We let the bidders compete in a $\alpha$-price auction: the highest bidder gets the object and pays a $\alpha$-convex combination of the first- and second-highest bid. The static Nash equilibria for all of these auctions are the same - both bidders choose $b_i=0.95$. The results are summarized in Figure \ref{fig:alpha}. For low values of $\alpha$ we observe the dominant strategy equilibrium, for high values we observe collusion, and intermediate values may lead to either outcome.

\begin{result}
\item The fraction of times our algorithms converge to the static-Nash equilibrium in $\alpha$-price auctions is increasing in $\alpha$. When $\alpha$ is close to $2$, they always converge to the fully-competitive outcomes and the fraction drops gradually for lower $\alpha$.

\end{result}

\begin{figure}[h]
\centering
\begin{tikzpicture}
\begin{axis}[ylabel near ticks,
    xlabel near ticks,
    xlabel={$\alpha$},
    ylabel={\%}, ymax=100,ymin=0, minor y tick num = 3, area style, colormap/jet,]
\addplot+[ybar interval,mark=no] plot coordinates { (1, 100) (1.1, 100) (1.2, 100) (1.3, 100) (1.4, 86) (1.5, 41) (1.6, 15) (1.7, 2) (1.8, 0) (1.9, 0) (2, 0)};
\end{axis}
\end{tikzpicture}
\caption{Percentage of simulations that converge on a collusive outcome for different values of $\alpha$.}
\label{fig:alpha}
\end{figure}

Notice that, except for $\alpha = 2$, none of these formats has a dominant strategy. For all $\alpha<2$ the best response to any bid less than $0.95$ is to bid one increment above. This result allows us to conclude that strategic simplicity is not the main driver of the observed differences between FPA and SPA.

\paragraph{Few profitable deviations.} The second hypothesis is that the difference is caused by the following property of the FPA and SPA. Suppose the two bidders bid  $(0.35, 0.35)$. In a SPA any bid above $0.35$ is a profitable deviation in the immediate future, because it simply doubles the expected return for the deviating bidder. In contrast, in a FPA, only bids above $0.35$ and below $0.65$ are profitable: above $0.65$ the expected profit of the deviating bidder is lower than with the bid $0.35$. It is possible that when our algorithms randomly experiment in search for a better strategy, the have a harder time finding one when the set of immediately profitable deviations is smaller. And as a result, they may ``get stuck.''\footnote{Moreover, the set of deviation bids that are profitable is increasing in $\alpha$, so this explanation could also explain the pattern we observed in  Figure \ref{fig:alpha}.}

To test this hypothesis, we restrict the domain of experimentation of our algorithms. We constrain the algorithms to local deviations: only bids immediately above or below are admissible deviations from the currently optimal strategy. When experimentation is only local, the difference in the shape of the payoff is reduced, and particularly the difficulty in finding the profitable deviation is eliminated. Now for both SPA and FPA one out of the two possible deviations is profitable and one is not.

We present the results in Figure \ref{fig:local} in the Appendix. The results do not change substantially: there is slightly less coordination on the dominant strategy in SPA and slightly lower bidding in FPA, but the general pattern remains.

\begin{result}
\item Modified algorithms that experiment only locally converge to approximately the same outcomes as in Result \ref{Result1}: static-Nash equilibrium bidding in SPA and much lower bids in FPA.
\item In other words, our results are robust to changing the experimentation method of the algorithms from global to local.
\end{result}

\paragraph{Collusion is hard to discover?} Another possible explanation is that the different nature of the games makes it harder for the algorithms to discover tacit collusion in the SPA than in the FPA. We chose to use optimistic initialization of the algorithms that leads to a considerable period of exploration. It is possible that this phase, rich of uncertainty, prevents the bidders from finding a good collusion outcome in SPA. To test this hypothesis, we initialize the algorithms at a collusive outcome. Naturally, if the experimentation parameter $\epsilon = \varepsilon e^{-\beta t}$ is too low, the algorithms never leave their initialization. However, we find that with enough experimentation, the FPA remains collusive (not necessarily in the outcome it had been initialized to), while the SPA reverts back to dominant strategy.

\begin{figure}
    \centering
    \includegraphics[scale=0.65]{collusive_init.png}
    \caption{The picture shows the moving average of the winning bid in a single simulation for both FPA and SPA initialized with a bias towards collusion at bids of $(0.4,0.4)$. }
\label{fig:initialization}
\end{figure}

Figure \ref{fig:initialization} shows that with exploration parameter $\epsilon = 0.25 e^{-0.0002 t}$ the second-price auction overcomes its bias towards collusion, reverting back to perfect competition. The first-price auction remains collusive instead.

\paragraph{Incentives to deviate to near-by Bids.}
The final difference we explore is that in SPA when a bidder finds a profitable deviation by bidding more than the opponent, every higher bid is equally profitable. In contrast, in FPA when a bidder deviates to a higher bid, even if it is profitable (and as we pointed out before, not all higher bids are profitable in FPA), the bidder should learn that lowering their bid to just above their opponent is even better. As a result, when bidders get outside a temporary coordination at equal bids, the next time they start bidding the same amount, the bids tend to be lower in FPA than in SPA. This helps the bidders converge to a local cycle at low bids instead of getting stuck at the static Nash equilibrium. Our simulations show the most support for that economic force.

To test this intuition, we introduce an artificial push towards lower bids in the exploration rule. Once the agents reach the steady state, with probability $1-\chi$ they behave according to the $\epsilon$-greedy policy. However, with probability $\chi$ they will choose the lowest bid whose value is ``close'' to the value of the current strategy.\footnote{More precisely, they choose the lowest bid such that the value of that strategy lies within $d = 0.3$ of the value of the current optimal strategy. This translates to an artificial force towards lower bids.} The results of this simulation are summarized in Figure \ref{fig:pushdown}.

\begin{figure}[h]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{pushdown_FPA.png}
  \caption{FPA bids}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{pushdown_SPA.png}
  \caption{SPA bids}
\end{subfigure}
\caption{Frequencies of bids with a downward force in the exploration rule. $\chi = 0.62 e^{-0.002 t}$.}
\label{fig:pushdown}
\end{figure}

\begin{result}
Algorithms designed to explore low bids tend to converge to below-static Nash equilibrium bids in both first and second-price auctions.
\end{result}

As is clear in this Figure \ref{fig:pushdown}, there are still differences between the FPA and SPA, but that is to be expected since there is no way to replicate the same exploration dynamics in the two games. Yet, these results are highly suggestive that this tendency towards local deviations and hence re-coordination on near-by bids is an important force for the difference in outcomes between FPA and SPA. It also explains why in the $\alpha$-price auctions collusive prices slowly emerge as $\alpha$ decreases.

\subsection{How Collusive Prices are Supported}

To understand our results, it is natural to ask how are the low prices maintained in long-term play. In fact, the algorithms do not explicitly try to learn dynamic strategies, such as the ones from our analysis of repeated games, mainly because our algorithms do not keep track of the history of the game --- other than via the $Q$ matrix, which is a very coarse way for keeping history. Moreover, our algorithms are designed to take the best action based on the estimates of long-term payoff consequences and hence they are not designed to execute punishment strategies in case their opponent deviates.

Admittedly, describing and understanding precisely the evolution of two Q-learning algorithms with 19 different actions each is hard. Based on the analysis of the different simulations and the theoretical work of \citet{Banchio2022b} we have built the following intuition.

Suppose current estimates of $Q$ are such that both bidders choose to bid the same low amount $b$. If there were many periods without exploration, bidders' estimates of $Q(b)$ would converge towards $Q(b)=\frac{1-b}{2(1-\gamma)}$. As a bidder $i$ explores to other bids, it is going eventually observe that a bit higher bid, $b'$, yields on average higher payoffs (since it doubles the probability of winning at a slight increase in payment). That is, eventually $Q_i(b')$ overtakes $Q_i(b)$ and bidder $i$ switches to $b'$ in most periods. That is not stable: the opponent $j'$ estimate of $Q_j(b)$ starts decreasing as a result. At some point it becomes sufficiently low that $j$ starts switching to other bids too. If they switch to $b''<b'$ they will continue losing and reducing $Q_j(b'').$ If they switch to $b'$ they start winning on average half of the time and it may become a new stable profile of bids for a while. If they switch to bidding $b''>b'$ then the instability we reset and now bidder $i$ will after a while start searching for a more profitable bid.

The key is what happens if they luckily re-coordinate temporarily at $b'$. Now, for bidder $i$ the estimate of $Q_i(b')$ will start decreasing (because they win as often as before at bid $b$, just pay a bit more). The algorithm is agnostic over why that is happening and hence will try to go back to $b$ that had a higher long-term payoff. The same happens to player $j$, hence at some point they will both bid $b$. Additionally, if player $j$ manages to experiment to a lower bid and wins, the players may go back to both bidding $b$ even quicker. If exploration never dies, this cycle can continue forever and include multiple bid levels (it has to include multiple ones).

This is the behavior we observed in our simulation with $100,000,000$ periods and exploration that never stopped (described in Figure \ref{fig:exp}).

At the essence, two forces affect the FPA equilibrium: one directs bids upwards and the other steers them downwards. In a SPA the latter disappears, leaving only an upward force which naturally leads to convergence on the highest possible pair.

A slightly different way to see the intuition is that when the players get to the point of both believing that $Q(0.95)$ is the highest and one starts experimenting to lower bids, in SPA there is no reason for the other bidder to ``follow.'' In contrast, in FPA there is a good reason to go down - not to match, but to be just one bid increment above. This makes it easier in FPA than in SPA for the players to escape $(0.95,0.95)$ (or any other high bids) and maintain those lower bids.

This intuition helps understand two more results. First, in the next section we discuss what happens in FPA if the auctioneer gives at the end of each auction additional information about the bids of others. Second, in the section with extensions it helps us resolve why the tacit collusion results in revenues that are meaningfully higher than $0.05$ or $0.10$.

\section{Auction Design.}\label{sec:design}
The results presented so far assume that the designer provides the bare minimum information to the bidders: after each auction she reports solely whether they won or not. However, the designer has access to a wider set of messages: for example, she might want to communicate what the highest bid was, or an anonymized distribution of bids. In this section we show how these design considerations may have a profound impact on the outcomes of play by the bidding algorithms.

We focus here on first-price auctions to see if the ''collusion'' can be reduced via such a policy. Such auction design policy also has clear practical relevance.

One of the consequences that we now consider is that when an algorithm is provided with information about the highest opponent bid after the auction, it can estimate its payoff in hindsight. That is, it can calculate the counterfactual payoff for every possible bid, not only the one chosen in the auction. This information allows the Q-learning algorithm to learn about all bids contemporaneously.

Our simulations show that with such change in information policy, if our algorithms take that information into account, collusion disappears. In Figure \ref{fig:synchronous} we present the result of simulating a FPA with \emph{synchronous} updating Q-learners. Formally, the algorithms update the action-value function for all entries, using the return in hindsight $R_t(a)$ for each action:
\[Q_{t+1}(a) = (1-\alpha)Q_t(a) + \alpha\big[R_t(a) + \gamma \max_{a'} Q_t(a')\big] \qquad \forall a\]

\begin{result}
The synchronous algorithms (that update $Q$ in hindsight based on the information about the highest competitor bid) in FPA result in substantially higher revenue (close to the static Nash equilibrium) than the asynchronous algorithms described in \ref{Result1}.
\end{result}

\begin{figure}
    \centering
    \begin{subfigure}{.5\textwidth}
        \centering
        \includegraphics[width=83mm]{FPA_synchronous.png}
        \caption{Frequencies of bids}
    \end{subfigure}%
    \begin{subfigure}{.5\textwidth}
        \centering
        \includegraphics[width=83mm]{cycles.pdf}
        \caption{Cycling behavior}
        \label{subfig:cycles}
    \end{subfigure}
    \caption{Outcomes of 500 simulations obtained with the standard set of parameters but synchronous updating.}
    \label{fig:synchronous}
\end{figure}

The difference in outcomes in Figures \ref{fig:base} and \ref{fig:synchronous} is clear: the algorithms are now competing almost perfectly.\footnote{Competition remains stuck in $(0.9,0.9)$ because of the discretization of the bid space. Finer discretizations approximate competition with higher and higher precision.} This result is consistent with the findings of \citet{Asker2022} in a Bertrand pricing game, where synchronous algorithms compete prices down to marginal cost while asynchronous algorithms collude on supra-competitive prices.

We can rationalize these outcomes through the mathematical apparatus of \citet{Banchio2022b}. When applying their framework, the synchronous algorithm always suffers from regret towards higher actions, even though the regret is mitigated by the reduction in profit associated with increasing one's bid. The highest regret bid is the one directly above the winning bid: that will be the next preferred action. This process slowly brings the optimal action to the maximum, as shown in Figure \ref{subfig:cycles} that illustrates bid-paths reminiscent of Edgeworth's cycles.\footnote{Such price/bid cycles have been often documented in algorithmic pricing and auctions (see for example \citet{Musolff2021}, \citet{Edelman2007}).} Experimentation forces the algorithms to try a dominated action, and sometimes this leads to both bidders coordinating on lower bids. However, the regret process described before soon brings both bidders back to the competitive state.

A related intuition is related to what we discussed above: For re-coordination on low bids it is important that when bidders switch to $b'>b$ and then coordinate on $b'$, over time they understand that $b$ was actually better. But with synchronous learning they update their estimate of the value of $b$ conditional on their opponent playing $b'$: their value for $b$ is bound to fall. The only hope for recoordination are second-order events, when both players experiment at the same time.

The analysis just carried out turns out to be an important design consideration in online advertising. Recently, Google changed their ad auctioning system from a SPA to a FPA. Alongside this change, bidders now observe the highest bid of their competitor. This fact squares perfectly with the intuition we obtained from our simple simulations: the ability to compute regret introduces an incentive to outbid the opponent that is missing otherwise. This is strong evidence that design choices such as communication and information may have a large impact on outcomes and consequently on revenues.\footnote{See \citet{dworczak2020mechanism} for other reasons why information disclosure after an auction can affect revenues and efficiency.}

\section{Extensions.}
One puzzle remains open, when observing the results in Figure \ref{fig:baseFPA}. While collusion is effective at improving the bidders' payoffs, it is imperfect: why are the bidders colluding on average on $(0.3,0.3)$ instead of maximizing their profit by coordinating on $(0.05,0.05)$? The explanation turns out to be rather mechanical. In fact, if the algorithms were choosing a price at random, then a bid of $0.5$ would be profitable $50\%$ of the time, while a bid of $0.05$ only $5\%$. In order to collude on bids of $0.05$, the two algorithms would need to each bid $0.05$ repeatedly and split the surplus, which is rather unlikely. To test this theory, we run an experiment with additional negative bids, between $-0.3$ and $0$. These bids can be thought of as non-participation options. None will clearly ever be chosen in equilibrium, but now, when bidders are choosing at random, a bid of $0.05$ is profitable almost $30\%$ of the time. The results of this experiment are shown in Figure \ref{fig:nobid}

\begin{result}
The algorithms find it easier to coordinate on the lowest bid in a first-price auction when they are given the option to not participate.
\end{result}

\begin{figure}[h!]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{nobid_FPA.png}
  \caption{FPA bids}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{nobid_SPA.png}
  \caption{SPA bids}
\end{subfigure}
\caption{Frequencies of bids excluding all the additional negative bids (simulations never converge to the negative bids).}
\label{fig:nobid}
\end{figure}
As it turns out, collusion now occurs more frequently on the most profitable pairs, validating our earlier explanation.

\paragraph{Reserve prices.}
From the standard theory of auctions we expect reserve prices to matter only when they are binding. In principle, then, the SPA should be unaffected and the FPA should be essentially unaffected as well as long as the reserve is lower than the collusion pairs. To test this, we run an experiment with a reserve price $r = 0.2$. The results are shown in Figure \ref{fig:reserve}

\begin{result}
With reserve prices, the algorithms coordinate on a distribution that appears skewed towards higher bids.
\end{result}

\begin{figure}[h]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{reserve_FPA.png}
  \caption{FPA bids}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{reserve_SPA.png}
  \caption{SPA bids}
\end{subfigure}
\caption{Frequencies of bids with a reserve price of $0.2$.}
\label{fig:reserve}
\end{figure}

These graphs conform well with our intuition: the two distribution of outcomes in the FPA with reserve is close to a truncation of the original  distribution. Note that this also relates to our previous observation: the bids below the reserve price act as ``non-participation'' bids, helping bidders coordinate on the most-profitable outcome. We will show that the introduction of a fringe of bidders instead has additional effects on the outcome distribution.

\paragraph{More competition.} Our model so far has been dealing with 2 bidders competing against each other. The results hold with 3 bidders as well, as shown in Figure \ref{fig:three}.
\begin{result}
With three bidders, in a first-price auction the algorithms are more likely to converge on collusive outcomes for higher values of the discount factor. In the second-price auction the outcome remains the static Nash equilibrium.
\end{result}
However, note that for the discount factor used so far ($\gamma = 0.99$) collusion is harder to sustain with three bidders. If we run the experiment with $\gamma = 0.999$ however, collusion is restored.

One further way to add competition is to add a fringe of non-strategic bidders. We model the fringe as an additional bid randomly drawn from a uniform distribution over the unit interval. The outcomes of the experiments are presented in Figure \ref{fig:fringe}.

\begin{figure}[h]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{fringe_FPA.png}
  \caption{FPA bids}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=83mm]{fringe_SPA.png}
  \caption{SPA bids}
\end{subfigure}
\caption{Frequencies of bids with a Uniform$(0,1)$ fringe.}
\label{fig:fringe}
\end{figure}

\begin{result}
With a fringe of bidders with bids drawn from a uniform $[0,1]$ distribution, the bids in the first-price auction increase over the optimal best-response. In the second-price auction the outcome remains the static Nash equilibrium.
\end{result}

In particular, notice how in the FPA the bids go up, mostly above the optimal collusion above the fringe. Given the values of the experiment, the optimal bid for both agents would be $0.5$. Instead, we observe a distribution that concentrates the most mass on $0.6$ and $0.65$, thus reducing the gains from collusion significantly.

\section{Conclusions.}
We have analyzed simple auctions where bidders with fixed values compete repeatedly in many auctions. Instead of using Nash equilibrium analysis, we asked how play would evolve if two (and sometimes more) bidders used simple artificial-intelligence algorithms. We simulated simple Q-learning algorithms and have shown a remarkable difference in the performance of first-price and second-price auctions.

This difference points to a new intuition about the play of such algorithms in real-life settings. Our previous intuitions are built either on the expectation that simple algorithms could converge to the static Nash equilibrium (and then, our Nash equilibrium analysis would predict no significant difference in the revenues), or on the expectation that these algorithms will coordinate on a tacit-collusive equilibrium of the repeated game. In the latter case, the theory of repeated games again would not predict differences between the first and second-price auctions. As we discussed, the incentive compatibility constraints for collusive strongly symmetric equilibria are approximately the same in those two formats (they are different only because we assume bids have to be chosen from a grid). Moreover, when one considers asymmetric equilibria (like bid rotation, where bidders alternate who wins), tacit collusion is much easier to sustain in repeated second-price auctions.

We show that a new dynamic force creates the observed difference in outcomes: in a second-price auction any deviation to a bid higher than the competitor's bid is equally profitable. In the first-price auction, the most profitable deviation is by just one bid increment more than the opponent. As a result, when because of experimentation bidders deviate from equally low bids, the losing algorithm starts exploration that can only end (at least temporarily) when both bidders re-coordinate and believe that winning half of the time at this bid is better (for long-term payoffs) than deviating to a higher bid.
Such re-coordination happens at lower bids in the first-price auction than in the second-price auction, resulting in different behavior in the long run.

We note that the seeming convergence to low bids is qualitatively different than the convergence to the static Nash equilibrium. For any fixed action of an opponent, our algorithms learn to best respond. Hence, if they converge to a fixed profile of actions then it must be a static Nash equilibrium. This is what happens in the SPA and in the FPA when the auctioneer provides them information that facilitates synchronous learning --- estimating the value of all bids in every round.
However, without that information, in the first-price auction the bidders never properly converge: intuitively they end up in local cycles: for example, they get to bidding 0.3 each, then one of them experiments and they learn that a higher bid is more profitable - they may learn that 0.35 is a best response - and then a short phase of experimentation takes place, where the opponent tries different actions to counteract the streak of losses. After re-coordinating on (0.35, 0.35), over time they learn that their average payoff is worse than the payoff from 0.3, and they try to get back to 0.3. With some luck, they both try lower bids at the same time and they learn that indeed that is a better strategy and switch back to 0.3 (or to even lower bids). In other words, especially when we look at the system without taking the experimentation parameter $\epsilon$ to zero, the two players spend most of the time on the  diagonal (with
equal bids), but do not settle in one place forever. Instead they move up and down.

If the auctioneer provides them with information about the lowest bid to win, when they move from the bid of 0.3 to 0.35, they synchronously learn that while 0.3 used to give a better payoff than 0.35, given the current state of the system, 0.35 is actually better than 0.3, and the incentive to go back to 0.3 disappears. That explains why providing this additional information (if it is not ignored by the algorithms or used in some other way than in our simulations) leads to more competitive bidding.

Many questions remain open. First, one may be worried about the robustness of our findings to allowing other artificial intelligence algorithms to play these games. We expect that the new force we have identified will be present in many algorithms that operate with limited information (for example, in first-price auctions, without observing bids of others).
Related to that question of other algorithms is what would happen if we made the Q-learning algorithms more sophisticated, for example, by keeping as a state whether the bidder won or lost the last auction (or what fraction of auctions they have won in the last minute).

Second, we have considered the policy of revealing additional information in first-price auctions by allowing the algorithms to update the Q vector synchronously but otherwise we kept the algorithm unchanged. A realistic concern rooted in economic theory of repeated games is that providing additional information could facilitate tacit collusion - bidders could switch to algorithms that keep last two bids in short-term memory as states and estimate a Q matrix, with each vector representing a different pair of recent bids. If so, providing information could backfire by facilitating instead of discouraging tacit collusion (for example, in the form of bid rotation).

Third, we looked at a very simple environment with two symmetric bidders and fixed values. While we have shown robustness of our findings to the introduction of a third bidder and a competitive fringe, many questions remain open. We think the most interesting question is how asymmetry would affect the findings (asymmetry in values and/or algorithms). Time-varying valuations are also of great interest since they would provide additional rationale for using artificial intelligence algorithms that constantly experiment and try to adapt to the changing competitive environment.

\bibliographystyle{ecta}

redpajama_set_name: RedPajamaArXiv


Schoology is a popular tool that is gaining much popularity in school systems. Now that there is an app for Schoology available on the iPad and other mobile devices, there are even more advantages to using Schoology in the classroom.
Now that our schools are getting access to new equipment for the classroom like iPads, this is a great time to start getting teachers and students on Schoology. Just like the desktop version, the app has all of the same features except it is in mobile form. If you are unfamiliar with Schoology in general, check out this video.
Wolfram Alpha is a website that allows easy access to a large amount of information. It has facts, data, and can be used to solve mathematical equations. For example, I can ask Wolfram Alpha for the "population of the united states". After this is interpreted, I receive a page full of data and statistics. This page may seem familiar if you have ever asked Siri on the iPhone for statistics, this is where Apple pulls that information from.
Quizlet is my hero. I have thrown away all my paper flash cards in exchange for this amazing studying tool. Whenever I have a test in any subject that requires memorization, I immediately turn to it. Honestly, I am not a firm believer in rote memorization as a form of education. But Quizlet is different and I always leave knowing my facts cold. The interface is clean and simple, the activities are engaging, and learning with Quizlet is, dare I say, fun!
My favorite aspect of Quizlet is its versatility. Many falsely assume that flashcards are merely for english vocabulary words, but that is far from the truth. I use Quizlet for every class on my schedule. For an upcoming history test, I will make a set that asks me questions that require me to reply with facts and dates. For a chemistry exam, I will use pictures of bonds and ask myself to identify them. For a spanish quiz, I will use the Speller activity to say the spanish word aloud in the correct pronunciation before I spell it correctly. Quizlet ingrains everything and anything I want into my mind.
Entering in the words and definitions becomes my first form of studying, as it forces me to type out what I need to know. After data entry, I can pick between a wide variety of learning activities, from a fast-paced space-race or scatter game to a slower-paced (but no less engaging) test or Speller.


text:
Measures to support Russian CSOs under sanctions
Home / Regional News / Measures to support Russian CSOs under sanctions
Measures to support CSOs under sanctions
There are 40 support proposals on the list. Measures include the introduction of temporary tax and customs privileges, the provision of subsidies and preferential loans, and measures to support CSO staff.
Deputy Prime Minister Tatyana Golikova is asking nine ministries to consider measures to support CSOs. Those asked to consider measures include the Ministry of Finance, the Ministry of Health, and the Ministry of Justice, together with the Bank of Russia, the Presidential Grants Fund, and the Civic Chamber. The result must be reported to the Government and the Presidential Council for the Development of Civil Society and Human Rights by 22 April. The letter is at the disposal of the Social Information Agency (ASI).
The head of the Human Rights Council, Valery Fadeev, sent the letter of proposed measures to Prime Minister Mikhail Mishustin. Deputy Prime Minister Tatiana Golikova received the letter from the government office.
On 21 March, The GRANI Centre for the Civic Analysis and Independent Research Foundation collected and sent a proposal on the support of CSOs to the government and the Duma. On 25 March, the Committee on Development of Civil Society, Public and Religious Organizations held its first meeting. The next meeting will be held in April.
Svetlana Makovetskaya, Director of The GRANI Centre, explained that the proposed measures would help create conditions for "increasing the social usefulness of CSO activities", create "additional incentives for non-state actors to support CSOs", reduce adverse consequences for affected organisations, and link measures to support CSOs with sectoral support for other entities.
For the children of CSO employees, the proposal seeks to give "places in playgroups and nurseries" and "to provide summer activities and healthcare." An additional measure may also be to simplify CSO employees changing their employment status to self-employed.
Source: https://www.asi.org.ru/news/2022/03/30/tatyana-golikova-poruchila-rassmotret-mery-podderzhki-nko/
Translated by Ysabelle Smith

redpajama_set_name: RedPajamaCommonCrawl


BBB Newsletter. Welcome to October!
Welcome to the October edition of the Beach Business Breakfast newsletter, connecting the region's businesses for our mutual benefit and success.
The more you and your employees truly understand your customers, the more successful your business will be. This article from the Harvard Business Review shows how digging down to understand what motivates your customers will reap rewards.
Small businesses are being targeted by cyber crooks via the production of fake invoices. Read this cautionary tale about a Perth car dealership that's lost out to the tune of $65k. It pays to be ultra vigilant.
Funda Finance is a national non-bank lender based here in Newcastle, and their mission is to help small business owners grow their businesses by providing tailor-made finance solutions, specially designed for SMEs.
Small businesses can need funding for many reasons, from fueling expansion plans or purchasing equipment or stock, to assisting with cash flow or covering tax bills.
And as Funda focuses on solutions-based lending, it is able to help those who may not currently meet standard bank loan criteria. Funda offers unsecured business loans between $5000 and $250,000 with fair and fast terms. Loan applications are assessed within 12 business hours, and there is no penalty for early repayment.
Funda client, Ben Donaldson, Founder of Magnetic Shoes Co, is one of those businesses that has been able to grow thanks to Funda.
After experiencing frustrating delays from a bank, with 60% of the new product on pre-order from Australia's leading building supplier Stratco, he was able to get funding through Funda to manufacture the product.
Ben accessed $80,000 through Funda within 12 hours and was able to turn the loan into $260,000 in revenue.
To get in touch with Funda visit funda.com.au, or alternatively call Wade Crawford on 0488 999 018.
A chance to enjoy a VIP Supercars experience!
Funda is currently running a Newcastle 500 V8 Supercars competition, giving all small business owners the opportunity to win a V8s corporate experience. Head to www.funda.com.au/supercars and leave your details to enter.
Some lenders are currently giving away FREE $ if you refinance.
We have access to most of these lenders – so you can sit back and relax while we do the work (and get paid by the bank).
You (or your friends, family or clients) will get a better interest rate, save $ on repayments, and benefit from the lenders' FREE $ for refinancing.
DO YOUR CLIENTS HAVE MORTGAGES?
If your clients have a mortgage and are paying 4% or more, we can often help them get a better deal, save them $, and on settlement give them $ to spend with you! So message me or see me at the Beach Business Breakfast to discuss what we can do together.
We'd rather give you $ to help us grow our business than give it to Facebook or Google. If you refer family, friends or clients we'll give you a $250 gift card when their loan settles – to keep or to share with them.
We are available 6am-9pm 7 days a week – before work, after work, on weekends.
Am I the key to your next SMSF property purchase?
*Terms and condition, fees and charges and lending criteria do apply.


CAPTION : Needless to say, Ki wouldn't have anything to do with me after that. Try as I might she ignored me.
FOOKER : To tell the truth, I didn't really blame her. I was a mess back then. Arrogant, rude... I smoked a lot and fell in with the wrong crowd.
CAPTION : And I had a problem. A really BIG problem..
VARIOUS : CHUG IT! CHUG IT!


text:
Calophasidia dentifera är en fjärilsart som beskrevs av George Francis Hampson 1909. Calophasidia dentifera ingår i släktet Calophasidia och familjen nattflyn. Inga underarter finns listade i Catalogue of Life.

Källor

Nattflyn
dentifera

redpajama_set_name: RedPajamaWikipedia


FELIX® As Good As It Looks Senior Cat Fish Selection in Jelly has been specially formulated to meet the dietary needs of senior cats. Our delicious recipes include irresistible flavours of trout, sardine, tuna and salmon to add even more variety to your cat's diet as he grows older. As your cat gets older he'll need a diet to suit his ageing lifestyle, which is why our mouth-watering fish selection is great for fish-loving senior cats. We include high sources of protein with antioxidants, vitamins and minerals that have been carefully selected to support your cat in his senior years.
Specifically designed for cats of age 7+ to help keep them healthy in their later years.
Packed with lots of mouth watering ingredients including trout, sardine, tuna and salmon - great for fish-loving felines.
100% complete and balanced nutrition with added vitamins and essential minerals to keep your cat supported.
Served in individual pouches for convenience.


Athletes in Action has announced they will be hosting a Junior and Senior Elite Overnight Basketball Camps from July 28 to August 3 and Canadian Mennonite University for ages 11-18. The camp will be led by local basketball players Wyatt Anders, Carriera Lamoureux and Faith Hezekiah.
More details can be found below and registration can go to https://athletesinaction.ca/basketball/#basketballcamps or questions can go to basketball@athletesinaction.com.


Véase también "Automóviles" En Otras Regiones: Sugerencias en regiones mas especificas relacionadas a Categorias dentro del Directorio Dmoz.
Osana : Verified Customer : Florida - 28-11-2018 I just want to say what an amazing job Luis Santiago did working with me on a long distance. My car was in PR and I was already in Florida, well he took car of every single thing, a very smooth process and always available to answer my questions..
CODIGO FISCAL DE LA FEDERACION. ULTIMA REFORMA PUBLICADA EN EL DIARIO OFICIAL DE LA FEDERACION: 31 DE DICIEMBRE DE 2000. Código publicado en el Diario Oficial de la Federación el jueves 31 de diciembre de 1981..
Wedding Gift Calculator The biggest chore for a wedding guest is to decide how much money to put in the envelope. Not all weddings are equal. Therefore every couple getting married does not deserve the same gift..


text:
Tanjong Pagar Railway Station Open House
It has been slightly more than three years and a half since the Tanjong Pagar Railway Station bade farewell to the last train. Although gazetted as a National Monument in 2013, the now silent station is but an empty shell and remains out of bounds to the general public save for special events.
Well not anymore. The second day of the Chinese New Year saw the beginning of the station being made accessible to everyone on public holidays. And when there is a chance to allow my monkies to be exposed to a part of Singapore's history, you can be certain we will be there.
Very cooperative models I have got, no?
The distinctive architectural style of the building is an ornamented Art Deco one - an eclectic style that combines traditional craft motifs with Machine Age imagery and materials. On its exterior are four pillars with large statues, each symbolising one of the then Malaya's four economic pillars, Agriculture, Industry, Commerce and Transport.
Inside the main public hall, the lobby features batik-styled mosaic mural panels filled with Malayan scenes: rice planting, rubber tapping, shipping activities, bullock cart transport, crop-growing and tin mining.
But there was always only going to be one place that the monkies were going to zoom to.
The railway tracks, of course!
How often can one get to walk - or sit on - an actual railway SAFELY? Heh.
It turned out that there were quite a number of other families and people who took the opportunity to drop by the station and snap photos against the scenic backdrop. The wifey and I, included.
One of the last remaining bits of heritage left in the Tanjong Pagar area, the station is truly a place that offers a serene surrounding that can no longer be found in modern Singapore. Like a place forgotten by time, the silent beauty of the station took us back to a time when Singapore was much simpler as the monkies had a field day exploring the many now-empty control brick rooms.
Personally, I would have loved it more if there were fringe activities such as a guided tour and a photo exhibition depicting the grand contributions of the station, so that families with young kids can learn more and partake in its history. Perhaps for future Open Houses?
But either way, it is definitely a great place to visit the historical monument for a leisurely stroll along the track with the family and obviously, take photos and more photos.
It is always important to allow places of historical significance like the Tanjong Pagar Railway Station to remain accessible to the general public, so that not only adults but kids too will be able to know more about its significance to Singapore during its 79 years of operation.
Well, the monkies sure had their fair share of fun.
And if you are thinking of bringing your kids down for a stroll, you do not have to wait until the next public holiday. The station will be open to public again TOMORROW (27 Feb 2015) for a flea market from 6pm to 11pm. Don't miss it!


text:
Jackpot City Casino offers a huge games suite with more than 400 online casino games to choose from. Games include table games, card games, slot games, video poker, progressive games and more.
There are more than 300 slots games that players can choose from, including classic (reel) slots, video slots, megaspin slots, online fruit slots (pub fruities) and progressive slots.
Slots Games at Jackpot City Casino
Jackpot City Casino is powered by the online casino software provider, Microgaming. Microgaming is known to offer massive games suites as is evidenced by the 400 online casino games and more than 300 online slots games that are offered at Jackpot City Casino.
In addition, they release new games, with an emphasis on slots games, on a regular basis.
One of the newest online slots games to be released at Jackpot City Casino is the Lord of the Rings slot game.
The game was released on September 2, 2010 and has proved to be very popular.
The Lord of the Rings Slot Game
The Lord of the Rings slot game is a five reel video slots game. Instead of having paylines, it offers 243 different ways to win. The Lord of the Rings slot game is based on the popular 2001 movie, the Lord of the Rings: The Fellowship of the Ring which is based on the novel by the same name written by J.R.R. Tolkien.
The Lord of the Rings slot game uses movie footage, the sound track and amazing graphics that will leave The Lord of the Rings fans in seventh heaven. The Lord of the Rings slot game features wild symbols, scatter symbols, free spins and a jackpot.
Coins sizes vary from 0.01 to 0.05 with the maximum bet per spin set at 600.
The Lord of the Rings slot game at Jackpot City features symbols of various characters from the movie. These symbols include Aragon, Arwen, Bilbo Baggins, Frodo and high valued playing cards – 10, J, Q, K, A. It also features wild symbols and scatter symbols.
The wild symbol in the Lord of the Rings slot game is the Lord of the Rings logo. It substitutes for all other symbols except for the scatter symbol or the Eye of Sauron in order to complete a winning combination.
The Eye of Sauron symbol is an expanding wild symbol that can be found on reel 3 and it substitutes for all other symbols except the scatter symbol to form a winning combination.
The scatter symbol in the Lord of the Rings slot game is the ring symbol. Three or more ring symbols trigger 15 free spins.
Fifteen free spins can be triggered in the Lord of the Rings slot game by three scatter symbols (rings) being spun. Players can win up to 1,900,000 coins during the free spins.
During the free spins feature, one reel is randomly selected to turn into wilds that replace other symbols. Fifteen more free spins can be triggered if three scatter symbols are spun.


Providing meals to those in need. We would not be able to do this without our tribe. Thank you.
A month long drive at Pure Fitness that helped raise funds, helping others get a fresh start. We would not be able to do this without our tribe. Thank you.
A one of a kind event at Pure Fitness to help raise awareness and funds for the AFN. We would not be able to do this without our tribe. Thank you.
A can food drive that delivered hundreds of pounds of goods for those in our local community. We would not be able to do this without our tribe. Thank you.
A school supplies drive that provided the children of Clinton Elementary with an abundance of paper, backpacks, pencils, crayons and much more. We would not be able to do this without our tribe. Thank you.


J Struct Funct Genomics. 2005;6(2-3):209-17.
Page R1, Deacon AM, Lesley SA, Stevens RC.
Joint Center for Structural Genomics, USA.
Currently, 119 high resolution structures of Thermotoga maritima proteins have been determined by the Joint Center for Structural Genomics (JCSG, www.jcsg.org). Sixty-seven of these were solved using the first implementation of the multi-tiered crystallization strategy at the JCSG for the efficient crystallization of large numbers of protein targets. Previously, we reported the analysis of all proteins crystallized using this multi-tiered strategy [Lesley, S.A. et al. (2002) Proc. Natl. Acad. Sci. USA 99, 11664-11669; Page, R. et al. (2003) Acta Crystallogr. D Biol. Crystallogr. 59, 1028-1037]. Here, we extend the analysis and describe the crystallization characteristics of those proteins that produced diffraction quality crystals, ultimately resulting in high resolution structures. First, we found that over 77% (52) of the crystals used for structure determination were produced directly from high-throughput coarse screens, indicating that less than one quarter of the crystals (15) required fine screening. In addition, as observed for the proteome screen [Page, R. et al. (2003) Acta Crystallogr. D Biol. Crystallogr. 59, 1028-1037], the majority of conditions that produced crystals for natively expressed proteins, whose structures have been determined, were distinct from those of their more extensively purified and selenomethionine-labeled counterparts. Finally, 99% of the proteins whose structures were solved crystallized in conditions contained in the JCSG Minimal Core Screen [Page, R. et al. (2003) Acta Crystallogr. D Biol. Crystallogr. 59, 1028-1037; Page, R. and Stevens, R.C. (2004) Methods 34, 373-389], a set of 67 conditions previously identified as those most likely to produce crystals of a diverse set of proteins, confirming its success for rapid identification of proteins with a natural propensity to crystallize.


Our mission at Heritage Hearing Care is simple: to provide excellent services for hearing impaired people and their families in the Upper Ottawa Valley. We are local women dedicated to "helping ears to hear". We are committed to this area, its communities, its stories and its story-tellers. For us that means listening to you and helping you to listen to others.