text:
Open banking will change everything…eventually
New regulation in Europe and the UK has the potential to transform the face of financial services. Treasury Today finds out what the future might look like under PSD2 and open banking and what it all means for treasurers.
"A decade from now the banking environment will be unrecognisable," says Jonathan Turner, Partner, Financial Services at PwC, when asked what impact open banking will have on Europe's financial sector.
This is because the new rules, brought in by the European Union's Payment Service Directive 2 (PSD2) and the UK Competition and Markets Authority's (CMA) Open Banking initiative, is forcing banks to open up their IT infrastructures through application programme interfaces (APIs) to a host of regulated third parties – including their competitors. This may give rise to new business models and offer banks the opportunity to connect with the wider financial ecosystem and offer a host of innovative services to their clients.
Turner illustrates this point using the analogy of smart TVs. "When internet-enabled TVs first launched, some manufacturers allowed only one or two third parties to access them and provide services such as movie streaming," he says. "In many cases, these services eventually stopped working and many users were left with TVs that offered limited internet-enabled services."
However, as Turner explains, others allowed third parties to build and offer services through open APIs. Owners of these TVs were therefore able to access a variety of constantly evolving online services. Today, this is what most TV manufacturers offer.
"If you think about this in the context of banks, then I foresee there being some banks that attempt to continue operating much in the same way they have today," says Turner. "These will focus on innovating in-house and using their balance sheets to consume any competition."
"Then there will be those banks that actively encourage third parties to interact with their services through open APIs. It will be the latter that capture the imagination of customers and win the day, I believe."
Open banking in brief
Before exploring the impact that open banking will have on financial services, it is worth looking at what is behind the two key regulations driving it: PSD2 and the CMA's Open Banking initiative.
In 2015, the European Commission (EC) announced PSD2 to replace the original Payment Service Directive (PSD). Broadly speaking, PSD2 is intended to reflect the changes that have occurred in the payments industry because of digitisation. The aim is to make the European payments market more integrated, competitive, fair, cheap and efficient. PSD2 also sets out to improve security standards by regulating third-party financial service providers, known as account information service providers (AISPs) and payment initiation service providers (PISPs).
To achieve this, PSD2 has brought in a host of new rules that banks must comply with. The most impactful of these is the Access to Account rule, whereby customers can allow third-party AISPs and PISPs to access account data from their bank or banks through open APIs. The ASIPs and PISPs can then provide customers with a range of new services.
Launched in 2016 following a study of the retail banking landscape in the UK, the UK CMA's Open Banking initiative is similar. The regulation mandates the UK's nine largest current account providers to make customer data available to third parties. These third parties, with customer consent, can access this data to offer many different services including payment initiation and account aggregation payments. As in PSD2, this information will be shared through APIs.
With PSD2 going live in January this year, one of the most contentious issues has been the time taken to finalise and announce the regulatory technical standards (RTS). The RTS provides the rules that outline how banks should share account information, and how third parties can initiate payments under PSD2. These standards were eventually announced in late November 2017, with banks needing to comply by September 2019. In contrast, the standards for the UK's open banking were confirmed early on.
This delay has frustrated fintechs that have applied to become AISPs or PISPs. "Because it has taken so long for the technical standards to be published, the banks have rightly waited to invest significant sums into their first PSD2 offering," says James Higgins, Product Director at AccessPay. "This is frustrating for us at AccessPay as we have recently requested to become an AISP. We don't know what form the information we are requesting will be delivered in, making it hard to develop solutions for our customers."
Higgins also believes that PSD2 could have been more closely aligned with the developments in the EU's payments infrastructure. He uses the development of Australia's New Payments Platform (NPP) as an example of how it could have been done differently. "The NPP contains APIs that will allow a host of third parties to build overlay services using the rich data carried over the payment rails," he says. "The overall objective is like that of PSD2 – to drive innovation – yet it seems to be happening in a more aligned way, whereas in the EU it is all happening in silos."
The banks have likewise expressed their frustration about the delay in announcing the RTS standards. There is also some concern about the speed at which PSD2 is being implemented by EU countries into their law, with some countries already announcing that they will miss the deadline. "The goal is standardisation across Europe, but banks, in general, are still struggling to find the most optimal business models for themselves, the third-party providers and most importantly their customers," says Gunnar Berger, Head of Open Banking at Nordea.
What this means is that come 13th January 2018, the world will not have changed and we will not suddenly be able to access a host of new and exciting financial services, says Marcus Hughes, Director of Business Development, Bottomline Technologies. "In theory, PSD2 will be live, but in practice, the changes that will come with it will not be felt by the end-users of financial services until later," he says.
Some institutions are struggling with the burden of regulation. These banks see PSD2 as just one more thing they need to do.
Jonathan Turner, Partner, Financial Services, PwC
Banking impact
One of the most interesting aspects of open banking is the banking community's response to it. On the one hand, open banking poses a threat to incumbent banks and may see them lose parts of the customer relationships and, in some cases, revenue streams. On the other hand, it also provides banks with an opportunity to boost the products and services that they offer customers, potentially driving customer loyalty and creating new revenue streams.
A 2016 survey by Strategy& found that for the most part, banks in Europe were on the defensive when it came to PSD2 and open banking, with mostly negative perceptions of the new rules. Indeed, the study found that many saw PSD2 as just an exercise in compliance and a driver of cost, rather than an opportunity. Fast-forward a year and PwC's Turner believes that this remains the case for a handful of banks. "Some institutions are struggling with the burden of regulation," he says. "These banks see PSD2 as just one more thing they need to do."
Turner adds that it can be quite easy to tell which banks think that PSD2 is a burden or an opportunity by hearing about the main responsibilities of their Heads of Open Banking. "Some Heads of Open Banking focus solely on compliance," he says. "Others are focused on vision and creating the bank of the future."
Despite the reservations banks may have about open banking, many are experimenting in this area. The Strategy& survey found that 44% of banks planned to provide an open bank offering in the next five years. Moreover, 64% said they intended to integrate foreign products or functionalities into own digital offering. Forty-four percent also said they are planning to integrate their own products or functionalities into foreign ecosystems.
The bullish bank's view
Bullish about the open banking phenomenon, Nordea is keen to explore how it can leverage open APIs and work with fintechs to offer new services to its clients. "Nordea sees PSD2 and open banking as an opportunity to offer better services for our customers and to become more agile by partnering up," says Gunnar Berger, Head of Open Banking at Nordea. "By sharing the development load, we can bring innovative services to market faster. I believe many banks will follow the same approach and work to develop and leverage partnerships with future service providers. Those who lag behind in this area are missing a big opportunity."
Despite this positive outlook, Nordea is cognisant that change will not happen overnight. "There is still a lot of work we have to continue with within the bank," says Berger. "We must continue to update our core system. Equally important is changing the culture in the bank to improve how we work with external partners."
Overall though, Berger is excited about the future. "PSD2, along with changing customer behaviour and general technological developments, will have a big impact on the banking industry across Europe, leading to increased competition," he says. "We expect that PSD2 will boost innovation and the creation of new services by, or in collaboration with, third parties. This will lead to the emergence of a new payments ecosystem. After a time of maturation, we believe that corporate customers who experience the benefits of open banking will start to demand APIs and openness from their banking partners elsewhere in the world."
A brave new world for treasury?
PSD2 and open banking will clearly have a big impact on banks, but what impact will corporate treasurers, as the end users of financial services, see? At this stage, it is hard to say. For one thing, user cases are still developing, or have not yet been thought of. At the same time, the retail space is largely dominating the innovation agenda.
There are signs of what corporates can expect out there in the market, however. One area where treasury will most certainly see an early benefit is through aggregated account visibility. HSBC, for example, recently announced that it would enable UK customers to view their account balances from 21 different banking providers through their online banking platform.
Whilst this is aimed chiefly at retail customers, Bottomline's Hughes says that one of the most exciting aspects of open banking is that it brings efficient multi-banking to SMEs and middle market corporates. "Currently, only corporates on SWIFT can benefit from efficient multi-banking," he says. "Open banking and the use of APIs will bring the price of this down, meaning that smaller companies will be able to see up-to-date balances across their accounts in real-time and manage these through one portal." Hughes is also excited about what can be built on top of this foundation. "On top of multi-banking, solution providers can add sweeping, liquidity management, forecasting and the other value-add applications at a low price point to middle market corporates and SME," he says.
PwC's Turner adds that banks might even begin to offer SMEs and middle market corporates non-financial related products because of open banking. "There is no reason a bank could not offer CRM systems through its banking portal," he says. "In theory, it could even offer these companies a completely outsourced finance function, which is pretty revolutionary."
APIs: lessons from elsewhere
It can be useful to look at how APIs, which power open banking, are applied in various countries to see what else corporates might expect from open banking. In India, for example, the government has built a host of advanced API-enabled financial systems that banks can plug into. This has enabled banks to create new and exciting value propositions for their clients.
For example, Citi's Request to Pay Solution leverages India's Immediate Payment Service (IMPS) infrastructure and enables consumers to make real-time bank debits from their account using a tokenised address such as an email. For eCommerce companies selling in India, where card penetration is low, this solution opens the door to a much broader range of customers and gives them a low-cost collection method.
In China, the country which has arguably seen the biggest intermediation of financial services from technology companies, banks have been working with companies like Tencent and Alibaba to develop new collection tools for corporates. To provide one example, Standard Chartered has recently collaborated with Tencent's WeChat Pay to create a one-stop online collection solution for corporates in China. This allows eCommerce companies in China to reach over 800m consumers in China and benefit from improved reconciliation and payment processing.
Whilst it will take time for the new rules mandating open banking in Europe to take hold, these are clearly exciting times and a lot of change is expected over the coming years. It is unlikely that treasurers will need to make any major changes right away. However, by gaining a clearer understanding of their banks' open banking strategies, treasurers will be able to find out whether their providers are fully embracing the opportunities. And as new and innovative solutions become available in Europe, it will then be up to treasurers to understand how these solutions might add value to their businesses.


ULS Technology plc, through its subsidiaries, engages in the development and hosting of Internet based software applications for legal and property services businesses in the United Kingdom. The last earnings update was 142 days ago. More info.
ULS underperformed the Online Retail industry which returned 19.2% over the past year.
ULS underperformed the Market in United Kingdom of Great Britain and Northern Ireland which returned 2.1% over the past year.
Is ULS Technology undervalued based on future cash flows and its price relative to the stock market?
Here we compare the current share price of ULS Technology to its discounted cash flow analysis.value.
Below are the data sources, inputs and calculation used to determine the intrinsic value for ULS Technology.
The calculations below outline how an intrinsic value for ULS Technology is arrived at by discounting future cash flows to their present value using the 2 stage method. We try to start with analysts estimates of free cash flow, however if these are not available we use the most recent financial results. In the 1st stage we continue to grow the free cash flow over a 10 year period, with the growth rate trending towards the perpetual growth rate used in the 2nd stage. The 2nd stage assumes the company grows at a stable rate into perpetuity.
Amount off the current price ULS Technology is available for.
For ULS Technology to be considered undervalued it must be available for at least 20% below the current price. Less than 40% is even better.
ULS Technology's share price is below the future cash flow value, and at a moderate discount (> 20%).
ULS Technology's share price is below the future cash flow value, but not at a substantial discount (< 40%).
The amount the stock market is willing to pay for ULS Technology's earnings, growth and assets is considered below, and whether this is a fair price.
Are ULS Technology's earnings available for a low price, and how does this compare to other companies in the same industry?
** Primary Listing of ULS Technology.
ULS Technology is good value based on earnings compared to the GB Online Retail industry average.
ULS Technology is good value based on earnings compared to the United Kingdom of Great Britain and Northern Ireland market.
Does ULS Technology's expected growth come at a high price?
Unable to calculate PEG ratio for ULS Technology, we can't assess if its growth is good value.
What value do investors place on ULS Technology's assets?
* Primary Listing of ULS Technology.
ULS Technology is overvalued based on assets compared to the GB Online Retail industry average.
ULS Technology has a total score of 3/6, see the detailed checks below.
How is ULS Technology expected to perform in the next 1 to 3 years based on estimates from 1 analyst?
No analysts cover ULS Technology, future earnings growth has been estimated based on fundamentals.
Is ULS Technology expected to grow at an attractive rate?
ULS Technology's earnings are expected to decrease over the next 1-3 years, this is below the low risk savings rate of 1.2%.
ULS Technology's earnings are expected to decrease over the next 1-3 years, this is below the United Kingdom of Great Britain and Northern Ireland market average.
ULS Technology's revenue growth is positive but not above the United Kingdom of Great Britain and Northern Ireland market average.
ULS Technology's earnings are expected to decrease over the next 1-3 years, this is not considered high growth.
ULS Technology's revenue is expected to grow by 4.7% yearly, however this is not considered high growth (20% yearly).
All data from ULS Technology Company Filings, last reported 6 months ago, and in Trailing twelve months (TTM) annual period rather than quarterly.
Unable to establish if ULS Technology will efficiently use shareholders' funds in the future without estimates of Return on Equity.
ULS Technology has a total score of 0/6, see the detailed checks below.
How has ULS Technology performed over the past 5 years?
Below we compare ULS Technology's growth in the last year to its industry (Online Retail).
ULS Technology's year on year earnings growth rate has been positive over the past 5 years.
ULS Technology's earnings growth has exceeded the GB Online Retail industry average in the past year (120.7% vs 19.4%).
ULS Technology's revenue and profit over the past 5 years is shown below, any years where they have experienced a loss will show up in red.
ULS Technology has efficiently used shareholders' funds last year (Return on Equity greater than 20%).
ULS Technology used its assets more efficiently than the GB Online Retail industry average last year based on Return on Assets.
ULS Technology has improved its use of capital last year versus 3 years ago (Return on Capital Employed).
ULS Technology has a total score of 6/6, see the detailed checks below.
How is ULS Technology's financial health and their level of debt?
The boxes below represent the relative size of what makes up ULS Technology's finances.
ULS Technology's short term (1 year) commitments are greater than its holdings of cash and other short term assets.
ULS Technology's cash and other short term assets cover its long term commitments.
This treemap shows a more detailed breakdown of ULS Technology's finances. If any of them are yellow this indicates they may be out of proportion and red means they relate to one of the checks below.
All data from ULS Technology Company Filings, last reported 6 months ago.
ULS Technology's level of debt (51%) compared to net worth is high (greater than 40%).
The level of debt compared to net worth has been reduced over the past 5 years (64.1% vs 51% today).
Debt is well covered by operating cash flow (112.4%, greater than 20% of total debt).
Interest payments on debt are well covered by earnings (EBIT is 44.5x coverage).
ULS Technology has a total score of 4/6, see the detailed checks below.
What is ULS Technology's current dividend yield, its reliability and sustainability?
Current annual income from ULS Technology dividends. Estimated to be 3.57% next year.
If you bought £2,000 of ULS Technology shares you are expected to receive £69 in your first year as a dividend.
ULS Technology's pays a higher dividend yield than the bottom 25% of dividend payers in United Kingdom of Great Britain and Northern Ireland (2.01%).
ULS Technology's dividend is below the markets top 25% of dividend payers in United Kingdom of Great Britain and Northern Ireland (5.14%).
ULS Technology has been paying a dividend for less than 10 years and during this time payments have been volatile (annual drop of over 20%).
Dividend payments have increased, but ULS Technology only paid a dividend in the past 4 years.
What portion of ULS Technology's earnings are paid to the shareholders as a dividend.
ULS Technology has a total score of 2/6, see the detailed checks below.
What is the CEO of ULS Technology's salary, the management and board of directors tenure and is there insider trading?
Mr. Stephen Goodall, also known as Steve, has been Managing Director of ULS Technology plc since May 2, 2017 and Chief Executive Officer and Director since April 4, 2018. Mr. Goodall has been responsible for the day to day management of the Company's products and services. In addition, he has been instrumental in building the Company's success in tailoring conveyancing services and technology for lenders as well as introducing and commercialising new products and services for existing and new B2B relationships. He has worked for Legal & General Surveying Services (LGSS) for over 15 years, holding the post of Managing Director for 13 years. During his tenure, he successfully transformed LGSS from a very small surveying business into the number one, market leading, property risk and valuation distribution business, which in 2016 handled over 500,000 valuation instructions and generated revenue of approximately £80million. Prior to LGSS, he held key operations and service improvement roles in the clothing and printing industries, including the creation and growth of an offshore clothing company that ran a supply chain for blue chip retailers. He was a Director at Legal & General Surveying Services Limited and Legal & General Estate Agencies Limited. He was awarded the Royal Institute of Chartered Surveyors Fellowship in 2012 and also holds numerous high profile industry awards, both personally and on behalf of LGSS.
Insufficient data for Steve to establish whether their remuneration is reasonable compared to companies of similar size in United Kingdom of Great Britain and Northern Ireland.
The average tenure for the ULS Technology board of directors is less than 3 years, this suggests a new board.
ULS Technology has a total score of 0/6, this is not included on the snowflake, see the detailed checks below.
Investing in stocks comes with the risk that the share price will fall. … By comparing earnings per share (EPS) and share price changes over time, we can get a feel for how investor attitudes to a company have morphed over time. … During the unfortunate twelve months during which the ULS Technology share price fell, it actually saw its earnings per share (EPS) improve by 122%.
ULS Technology plc (LON:ULS): What Does Its Beta Value Mean For Your Portfolio?
If you're interested in ULS Technology plc (LON:ULS), then you might want to consider its beta (a measure of share price volatility) in order to understand how the stock could impact your portfolio. … Every stock in the market is exposed to this volatility, which is linked to the fact that stocks prices are correlated in an efficient market. … Beta can be a useful tool to understand how much a stock is influenced by market risk (volatility).
What does ULS Technology plc's (LON:ULS) Balance Sheet Tell Us About Its Future?
While small-cap stocks, such as ULS Technology plc (LON:ULS) with its market cap of UK£43m, are popular for their explosive growth, investors should also be aware of their balance sheet to judge whether the company can survive a downturn. … Online Retail businesses operating in the environment facing headwinds from current disruption,.
Is ULS Technology plc's (LON:ULS) P/E Ratio Really That Good?
Can ULS Technology plc's (LON:ULS) ROE Continue To Surpass The Industry Average?
If you're interested in ULS Technology plc (LON:ULS), then you might want to consider its beta (a measure of share price volatility) in order to understand how the stock could impact your portfolio. … Some investors use beta as a measure of how much a certain stock is impacted by market risk (volatility). … Any stock with a beta of greater than one is considered more volatile than the market, while those with a beta below one are either less volatile or poorly correlated with the market.
What Does ULS Technology plc's (LON:ULS) PE Ratio Tell You?
Is ULS Technology plc's (LON:ULS) 21.05% ROE Strong Compared To Its Industry?
2 Days Left Before ULS Technology plc (LON:ULS) Will Start Trading Ex-Dividend, Should You Buy?
ULS Technology plc, through its subsidiaries, engages in the development and hosting of Internet based software applications for legal and property services businesses in the United Kingdom. It provides software solutions that support online legal comparison services. The company also provides risk management and compliance consultancy services for solicitors and licensed conveyancers. Its solutions also cover mortgage lenders, builders and developers, intermediaries, estate agents, and energy performance certificates. The company, primarily, provides its services through white-labels to mortgage brokers, banks, building societies, and price comparison Websites. ULS Technology plc was founded in 2003 and is based in Thame, the United Kingdom.


text:
\section{Introduction}
\label{sec:introduction}
A large body of research across many disciplines seeks to draw inferences from data about the causal effect of a treatment. Examples include medical studies about effects of drugs on health outcomes \citep{bartroff2012sequential,tsiatiscovariateadj}, studies of impacts of advertising or marketing offers on consumer purchases \citep{lewis2011here,chan2010evaluating}, evaluations of the effectiveness of government programs or public policies \citep{heckman1991randomization,athey2016econometrics}, and more recently in online experimentation under the name ``A/B tests" (large-scale online randomized experiments), used widely by web facing technology firms \citep{Kohavi:2009, Kohavi:2013, googlesurvey, bakshy2014designing, gomez2016netflix}.

Historically, many works on digital experimentation have been focused on trustworthy analysis of the average treatment effect (ATE), where trustworthiness includes unbiased identification of ATE as well as correct assessment of its uncertainty (confidence interval, Type-I error and False Discovery Rate, sequential test, etc.) \citep{Kohavi:2013, DengBayesAB, johari2015always}. It is commonly understood among researchers and practitioners that individual treatment effects may differ in magnitude and even have opposite direction. This is called heterogeneous treatment effect (HTE) \citep{athey2015recursive, taddy2016nonparametric,tian2014simple}. There are two distinct reasons why understanding HTE is important.
\begin{compactenum}
   \item Personalized treatment: When there are many different potential treatments, e.g. policies, drug dosages, etc., we want to find the best treatment that maximize a pre-specified metric or utility for each individual. To achieve that, we will need to be able to predict treatment effect for every candidate treatment on every potential individual.
   \item Improvement and agile iteration: For a promising treatment with positive average effect, we want to find areas where the treatment can be further improved. One obvious direction is to identify scenarios where the treatment performs relatively bad, or even shows negative effects. Understanding HTE leads to sub-populations representing areas-to-improve.
\end{compactenum}
The first point put more emphasis on prediction and can be treated as a kind of supervised learning problem \citep{tian2014simple, athey2015recursive, wager2015estimation, taddy2016nonparametric}. The second point focus heavily on interpretation and summarization. The structure of HTE, i.e. how treatment effects differ between individuals, is of great interest as compared to only the value of individual treatment effect. In fact, even for the purpose of personalized treatment, it is also desired to have a interpretable and memorable result \citep{athey2016econometrics}.

In this paper, we aim at analyzing the heterogeneity of treatment effects in A/B tests, where the understanding of HTE structure is more important. This is because the goal of A/B tests is to improve a software, server side or client side. Code maintenance can be expenseive if we were to keep different branches of the same code just to achieve personalization. In reality, it is much easier to improve one version and make it work better for all users than maintaining ensembles of code. In addition to carving out the structure of HTE using an interpretable model, we also strive for a \emph{concise summarization} so it is memorable and actionable for product owners and engineers. Because there are many ways of defining sub-populations, we adopt the following definitions through out the paper.
\begin{defi-no}
A \textbf{covariate} is an observed attribute that can be used to split the population into sub-populations, e.g. browser, device, location, etc. A covariate can be continuous or discrete. A discrete covariate consists of a number of \textbf{levels}, e.g. different browser types. The number of levels for a covariate is called its \textbf{size}. A discrete covariate is often called a \textbf{segment type} with each of its levels called a \textbf{segment}. Discrete covariate can be either ordered, e.g. date, income levels, or categorical without ordering.
\end{defi-no}

We provide two concrete problems to illustrate the type of HTE we care about. The first one is a typical application in A/B testing and the second one is an extension outside of controlled experiments.
\begin{problem}\label{pb:1}
Suppose a treatment showed positive treatment effect on all browsers except several old versions of Internet Explorer for a period of two weeks. In addition, during the second week the treatment effects for Android devices dipped by a large amount. In this case the HTE can be decomposed into two pieces of information. First, this treatment does not work well for old IE browsers, possibly due to lack of legacy browser support. Second, something changed between the first and the second week for Android devices. Based on these two pieces of information, engineers can then prioritize work items to improve the legacy browser support and also works with product analysts to understand whey Android devices behaves different for the second week.
\end{problem}

\begin{problem}\label{pb:2}
We are monitoring a vital KPI metric of a business and observed an unexpected drop between this week and previous week. Since there is no sudden change in our product, we believe the drop is not systematically across the board, but rather due to some small sub-population, e.g. a set of mobile devices, geo-locations, etc. The task is to figure out these sub-populations that explain majority of the change so we can zoom in and fix the problem.
\end{problem}
The second example is a diagnosis problem. Although the pre-post comparison is not a controlled experiment and the difference between the two periods could be due to differences of many confounding factors, we are interested in identifying subgroups that defines the structure of heterogeneous pre-post differences. The problem of HTE structure and heterogeneous pre-post difference structure are essentially the same.

A common characteristic of the two problems above is that in both cases we want to summarize HTE structure in a very concise way. The obvious challenge is that the dimension of covariates is large and they can be further combined together to create higher order population segmentation. An ideal result is to identify lower order effects, i.e. sub-populations defined by small number of covariates. This is because product owners can only afford deeper investigations into a small number of cases. To make our result actionable, we have to bet in Sparsity-of-effects principle \citep{box2005statistics} since results with too complicated HTE structure will confuse people and discourage action taking.

This paper makes the following contributions:
\begin{compactenum}
   \item We propose using total-variation regularized regression (TV Regression) to produce concise summarization of HTE and the diagnosis problem. Our method can be applied to all discrete covariates for both categorical and ordered. We describe and implement an ADMM algorithm to efficiently solve the convex optimization problem.
   \item When sizes of covariates differ a lot, we allow penalizing total variation of different covariates differently. We provide a pre-tuning algorithm to help us find the best weights and our algorithm naturally generalizes to other problems such as Grouped Lasso \citep{yuan2006model}. We propose a post-screening stage that can be optionally added and further improve the result of TV Regression.
   \item We applied TV regression to both synthetic data and real experiment data. We also demonstrate the necessity to use weight pre-tuning.
\end{compactenum}

The rest of the paper is structured as follows. We first describe the background and existing approaches in literature to motivate our method. In Section~\ref{sec:TVregularized} we introduce Total-Variation regularized regression to achieve simultaneous HTE prediction and concise summarization. We demonstrate the effectiveness of TV regression using both synthetic and real experiment data in Section~\ref{sec:examples}, as well as the importance of adaptive between-graph weights. We also included a real application of TV regression for a diagnosis problem.

\section{Background}\label{sec:setting}
Suppose we have $N$ \emph{i.i.d.}\xspace data points labelled $i = 1, 2, \dots, N$, each of which consists of a covariate vector $X^i = (x^i_1,x^i_2,\dots,x^i_D)\in \mathbbm{R}^D$, a response $Y_i\in \mathbbm{R}$ and a treatment indicator $W_i \in \{0, 1\}$. These covariates can be categorical, e.g. gender, market, and can also be ordered, such as user age group, user visit frequency, date and time, etc. We do not consider continuous covariates and assume they can always be discretized into ordered covariates. This will not affect the generality of our approach as discussed later. Following the potential outcomes framework \citep{Imbens:2015, Rosenbaum1983, Rubin:1974}, we posit the existence of potential outcomes $Y_i^{(1)}$ and $Y_i^{(0)}$ corresponding respectively to the response the $i$-th user would have experienced with and without the treatment, and define the treatment effect at $x$ as
\begin{equation*}
	\tau(x) = \operatorname{\mathbb{E}}[Y^{(1)} - Y^{(0)} | X = x],
\end{equation*}
where $(X,Y^{(1)}, Y^{(0)})$ has the same distribution of $(X^i,Y_i^{(1)}, Y_i^{(0)}),i=1,\dots,N$. Our goal is to give an estimation of $\tau(x)$, which is the regression of $Y^{(1)} - Y^{(0)}$ with covariate vector $X$.

This is a regression problem if $Y_i^{(1)} - Y_i^{(0)}$ are directly observed. However, we can only ever observe one of the two potential outcomes $Y_i^{(1)}$ and $Y_i^{(0)}$ for $i^{\text{th}}$ data point. Let the observed outcome be $$Y_i^{\textrm{obs}} = W_i Y_i^{(1)} + (1-W_i)Y_i^{(0)}. $$ There are in general two different views to deal with this problem in the literature.

The first approach is to model the counter-factual pair $(Y^{(1)}_i, Y^{(0)}_i)$. The classic parametric model is a linear model with interaction terms between treatment assignment indicator $W$ and covariates $X$:
\begin{align}\label{eq:lm}
    Y^{\textrm{obs}}_i = f(X^i) + W_i \times \left ( \mu + \beta^T X^i \right ) + \epsilon_i,
\end{align}
where $f(X^i)$ is usually also a linear function, $\mu$ is the constant effect and $\beta \in \mathbbm{R}^D$ is the coefficient vector that encodes the HTE as a linear function of $X^i$. When dimensionality is high, a $\ell_1$ regularization \citep{tibshirani1996regression} is typically employed to enforce sparsity and prevent over-fitting. This approach has a long history and has appeared in different variations in the literature \citep{tian2014simple,imai2013estimating,taddy2016nonparametric}. This approach also includes tree based method where two tree based models are fit to for regression of $Y^{(1)}$ and $Y^{(0)}$ respectively.

A second approach is to directly estimate $\tau(x)$. Given a covariate vector $X = x$, if we can find enough data points with exactly the same $x$ in the experiment, and because treatment assignment is completely random, a unbiased estimator of $\tau(x)$ is just
\begin{equation}\label{def:tauhat}
	\hat{\tau}(x) = \bar{y}^{(1)}(x) - \bar{y}^{(0)}(x),
\end{equation}
where $\bar{y}^{(1)}(x)$ and $\bar{y}^{(0)}(x)$ is the treatment average and control average on the subset of data points having the same covareate $x$:
\begin{equation*}
	\overline{y}^{(w)}(x) = \frac{\sum_{X^i = x, W_i = w} Y_i}{N^{(w)}(x)}, \quad N^{(w)}(x) := \sum_{X^i = x, W_i = w} 1.
\end{equation*}
This may not be effective when dimensionality $D$ is large and $N^{(w)}(x)$ may be small for some $x$. But in reality, under the assumption of Sparsity-of-effects \citep{box2005statistics}, HTE might only depends on only a few covariates in the vector $x$, we can imagine a neighborhood of $x$ will have the same $\tau(x)$. This leads to Nearest Neighbor based method and nonparametric smoothing of \eqref{def:tauhat}. \citet{athey2015recursive} and \citet{wager2015estimation} used decision tree based method to adaptively partition the covariate space to achieve simultaneous neighborhood matching (leaves in tree) and HTE estimation. Different splitting criteria were used to achieve slightly different goals. For example, \citep{athey2015recursive} suggested to build a CART like decision tree by maximizing between leaf variance so the result tree can be used for both prediction and also interpretable.

\begin{figure*}[!tb]
    \centering
    \includegraphics[width=0.95\textwidth,height=0.3\textwidth]{ctree3}
    \caption{Causal Tree for Example~\ref{ex:1}. Number in each leaf is the predicted treatment effect and percentages are proportions of the whole population. The decision tree fitted the HTE quite well, but hard to summarize and be memorized.}
    \label{fig:ctree}
\end{figure*}

\subsection{Issues of Existing Approaches}\label{subsec:issue}
For applications like Problem~\ref{pb:1} and \ref{pb:2}, we found none of the currently existing approaches provides satisfying results. Consider the following example.
\begin{ex}\label{ex:1}
There are 4 categorical covariates $X_1,\dots, X_4$. $X_1$ has 10 levels, each 10\%. $X_2$ has 3 levels, with proportion 20\%, 50\% and 30\%. $X_3$ have 5 equally proportion levels and $X_4$ have 4 levels with 10\%,  40\%, 30\%, 20\%. Let $X_i$$:$$j$ be the jth level of $X_i$. We then put a simple ground truth effect as this: $X_2$$:$$2$ has first order effect of -0.1; there exist a second order effect of 0.1 on $X_1$$:$$(4,5,6,7)$ $\cap$ $X_3$$:$$(3,4)$. In addition, there is a constant global effect 0.03. $10,000$ \textit{i.i.d.} treatment and control observations are simulated from $N(0, 0.1)$ distribution respectively with treatment effect added to treatment observations according to the prescribed design.
\end{ex}

The ground truth HTE structure for Example~\ref{ex:1} can be concisely summarized by two numbers: one first order effect and one second order effect (global effect is not of interest). Note that the second order effect is defined as a block structure. In practice, block structure are common building blocks for HTE because it stands for clustering. These clusters often have semantic meaning. For example, web browsers are naturally classified by rendering engines and versions, similarly mobile devices are clustered by brand and operational systems. Block-wise structure is also very easy to memorize and allow product owners to clearly define where to further investigate. Although in Example~\ref{ex:1} the first order effect is ``point-wise'', i.e. consist of one level only. In general they can also be clusters of different levels(segments) for each covariate(segment type).

The setup of lower order effects suggests classic linear model with $\ell_1$ regularization will work well because the model captures first and second order effects as its coefficients in the linear regression. For an appropriated tuned parameter, we showed that Lasso does recover all first order and second order effects. However it suffers from two issues. First it could not cluster block-wise second order effects together. Instead, it will output 8 different coefficients for all combinations of $X_1$:$(4,5,6,7)$ and $X_3$:$(3,4)$. Second, we found Lasso still tend to output false positives, i.e. non-zero coefficients which are known to be 0 in the ground truth. We provide more details in Section~\ref{subsec:comparelasso} and similar results were shown in \citep{tian2014simple}. The main issue of Lasso failing to discover block structures is expected because the model does not perform clustering or look for block structure. Tree based method, however, naturally produce block-wise constant prediction. We applied Causal Tree \citep{athey2015recursive} (without honest splitting) and the result is shown in Figure~\ref{fig:ctree}. We point out at least two issues with tree based method:
\begin{compactenum}
   \item Although single decision tree is interpretable. It is not concise. This is because it has fixed number of branches in each split, usually binary. It is good at showing higher order effect as each leaf is block-wise, but it does not summarize lower order effect well. To convince yourself, look at Figure~\ref{fig:ctree} and imagine how one can recover the simple ground truth structure represented by 3 lower order effects using this tree. Note that the prediction from this tree is good, just hard to summarize.
   \item Tree relies on pruning to avoid over-fitting. But pruning only controls tree size without prevent over-fitting in earlier stage of splittings. This problem can be alleviated by putting aside a test set and use test data performance to optimize splitting on training data. This is the idea behind honest splitting in \citep{athey2015recursive, wager2015estimation}. Others suggested using statistical testing based splitting criteria to avoid over-fitting in each splitting \citep{hothorn2006unbiased}. These modifications further complicate the procedure making it even harder to be used in practice on big data.
\end{compactenum}

To overcome shortcomings of both approaches, we propose a new approach. We chose to use an additive model, similar to Lasso to summarize HTE as a set of lower order effects. But instead of fitting this additive model with sparsity requirement for nonzero coefficients, we also want to simultaneously recover block-wise structures among non-zero coefficients. This is known as \emph{simultaneous clustering and regression} \citep{bondell2008simultaneous}.

\section{Simultaneous clustering and \newline regression by Total-Variation \newline regularization}
\label{sec:TVregularized}
The idea of simultaneous clustering and regression naturally follows the discussions in Section~\ref{subsec:issue}. We use an additive model with $\ell_1$ penalty on nonzero coefficients, and want those nonzero coefficients to form block-wise clusters so that we can easily summarize the structure. In both applications of HTE and diagnosis problem, we limit the type of covariates as discrete. This is because nonlinear effect of a continuous covariate is hard to define in an additive model but can easily be approximated by linear effect of a discretized and ordered covariate. When all covariates are discrete, an additive model can be written as:
\begin{align}\label{eqn:additive}
	\tau(x) = u_0 + \sum_{k=1}^K u_k(x)
\end{align}
where $u_0$ represents the constant global treatment effect and $\{u_k(x)\}_{k=1}^K$ are different kinds of heterogeneous effects. The constant term $u_0$ is not of much interest when studying the heterogeneous treatment effect. Additive components can be defined with one covariate (first order effect) or multiple covariates (higher order effects). A model with up to second order effects is written as:
\begin{equation}\label{eqn:secondordermodel}
    \tau(X_i) = u_0 + \sum_{d=1}^D u_d(x_d) + \sum_{1 \le d < f \le D} u_{d,f}(x_d, x_f),
\end{equation}
where $D$ is the number of covariates.

Given the additive model of $\tau(x)$, we can fit the model to predict $Y^{\textrm{obs}}$ as in \eqref{eq:lm}. When viewing the problem as predicting individual $Y^{\textrm{obs}}$s, we need to deal with a very big dataset, quite often millions of $Y$'s for a large scale online services. On the other hand, for any combination of covariates $X^i = x$, we observe sufficient statistics
$$
(\bar{y}^{(1)}(x), \bar{y}^{(0)}(x), \hat{\tau}(x), N^{(1)}(x), N^{(0)}(x)).
$$
Turns out, when we treat $y^{(0)}(x) = \operatorname{\mathbb{E}}[Y_i^{(0)}| X^i = x]$ as model free, minimizing squared error for all $Y^{\textrm{obs}}$ (divided by their variances) is equivalent to the following optimization problem\footnote{Derivation is straightforward but tedious. See Appendix~\ref{app:equiv}.}:
\begin{equation}\label{eqn:master}
    \min_{\tau} \frac{1}{2}\sum_{x} M_{e}(x) (\hat{\tau}(x) - \tau(x))^2,
\end{equation}
where
\begin{equation}\label{def:M}
    M_{e}(x) = \frac{N^{(0)}(x)N^{(1)}(x)}{N^{(0)}(x)\operatorname{\mathrm{var}}^{(1)}(x)+N^{(1)}(x)\operatorname{\mathrm{var}}^{(0)}(x)}
\end{equation}
can be viewed as the effective sample size for the noisy measurement $\hat{\tau}(x)$ of the treatment effect at covariate combination $x$ and $\operatorname{\mathrm{var}}^{(w)}(x)$ are conditional variances of $Y^{(w)}$ at $x$.

Solving \eqref{eqn:master} can be seen as trying to predict $\tau(x)$ directly, as in \citep{athey2015recursive,wager2015estimation}. For moderate number of covariates or small size covariates, the number of covariate combinations is much smaller than the size of the original data, e.g. 10 binary covariates generate only 1k combinations, thus reducing the complexity of the optimization problem. When the number of covariate combinations is large, we apply a pre-screening stage to prune out majority of irrelevant covariates. More details in Section~\ref{subsec:prescreen}.
To cluster these effects together, we add penalty $|u_k(x) - u_{k}(x')|$ for every pair of $(x, x')$ that we would like to cluster together if their effects are close. The latter $\ell_1$ penalty is called total variation regularization for which we give detailed definition in the next section. TV regression solves the following convex optimization problem:
\begin{align}
\min_{\tau} \quad & \frac{1}{2}\sum_{x} M_{e}(x) (\hat{\tau}(x) - \tau(x))^2 + \lambda TV(\tau) \nonumber \\
\text{s.t.}\quad &\tau(x) = u_0 + \sum_{k=1}^K u_k(x), \label{eqn:tvregreg}
\end{align}
where $TV(\tau)$ is a given total variation penalty and $\lambda > 0$ is a parameter balancing the approximating accuracy and the regularization. We didn't add $\sum_{k,x} |u_k(x)|$ in \eqref{eqn:tvregreg} because we can easily include these terms within $TV(\tau)$.

\subsection{Total Variation on Graphs}
\label{subsec:tvongraphs}
Total variation penalty was introduced in images denoising \cite{rudin1992nonlinear, meyer2001oscillating, osher2003image}, where it is well-known to outperform $\ell_2$-regularized estimators like Laplacian smoothing (and also Laplacian eigenmaps) for its ability to better preserve sharp edges and object boundaries. In our setting, we can view each covariates combination $x$ as a pixel in a high dimensional image with global effect as background image and lower order effects as the foreground. In statistics literature, similar ideas are called Fused Lasso, trend filtering and Generalized Lasso \citep{tibshirani2005sparsity, tibshirani2011solution, wang2015trend, petersen2015fused, petersen2015fused, petersen2016convex}. In most cases, covariates are assumed to be ordered and the $\ell_1$ penalties of total variation are applied to pairs of neighborhood points/pixels. In our applications, a lot of covariates are categorical without ordered topology. To unify the two cases here we define TV penalty on a graph.

Let $G = (V, E)$ be a graph with nodes labelled $V = \{1, 2, \dots, n\}$ and edges $E = \{e_1, e_2, \dots, e_m\}$. Let $u : V \to \mathbbm{R}$ be a function defined on $G$. The total variation of $u$ is defined as
\begin{equation}\label{def:tvgraph0}
    TV(u) = \sum_{e \in E} |u(e_1) - u(e_0)|,
\end{equation}
where $e_0$ and $e_1$ are the two vertices of an edge $e$.

Every covariate $x_d$ defines a graph, denoted as $G_d = (V_d, E_d)$, where vertices $V_d$ are all its levels. These graphs can either be fully connected or encoding topological information. For example, for covariates with order, we connect all the consecutive nodes; for weekdays or months, we also connect Sunday to Monday and December to January to make loops; for combinatorial covariates, we can simply have a complete graph or use extra domain knowledge. For second order effect involving two covariates, its graph can be defined by the tensor product of the two single covariate graph, i.e., $G_d\times G_f = (V_d\times V_f, E_d\times E_f)$ where the vertex set is the Cartesian product of $V_d\times V_f$ and edge set $E_d\times E_f$ is defined such that there is an edge between $(x_d, x_f)$ and $(x'_d,x'_f)$ if and only if $x_d\sim x'_d$ in $E_d$ and $x_f=x'_f$ or $x_d = x'_d$ and $x_f \sim x'_f$ in $E_f$. Higher order graph is defined similarly.

For any term $u_k(x)$ in the additive model \eqref{eqn:additive}, a naive choice is to define TV by Eqn.~\eqref{def:tvgraph0} for each term and then sum them up to define $TV(\tau)$. However, in many applications, the sizes of different covariates vary a lot. For example, gender has only 2 states, but product category has over hundreds. Simply putting equal weights to every edge as in~\eqref{def:tvgraph0} does not work well due to this heterogeneity. We define
\begin{equation}\label{eqn:tvforadditive}
	TV(\tau) = \sum_{k=1}^K w_k TV(u_k),
\end{equation}
in which $\{w_k\}_{k=1}^K$ are some to-be-determined weights. A first-order effect $u_d(x_d)$ has the following modified TV
\begin{equation}\label{eqn:tvfor1st}
	TV(u_d) = (1-\alpha)\sum_{e \in E_d} |u_d(e_1) - u_d(e_0)| + \alpha \sum_{v\in V_d} |u_d(v)|
\end{equation}
and a second-order effect $u_{d, f}(x_{d}, x_{f})$ has TV defined by
\begin{align}
	TV(u_{d, f}) & = (1-\alpha)\sum_{e \in E_d\times E_f} |u_{d,f}(e_1) - u_{d,f}(e_0)|  \nonumber \\
    &	+ \alpha \sum_{v\in V_d\times V_f} |u_{d,f}(v)|. \label{eqn:tvforhighorder}
\end{align}
Note that we've incorporated Lasso penalty $|u_k(x)|$ into TV penalty so we no longer need it in \eqref{eqn:tvregreg}. TV for higher order effects can be defined similarly, but we believe that first and second order effects are most useful for practice.

Our definition of TV on graphs has the following properties:
\begin{compactitem}
\item We define the TV separately for different terms because each term so we encourage effect clustering within each additive term. This separation also makes it computationally more efficient.
\item In Section~\ref{subsec:weights} we introduce a simple procedure for choosing weights $\{w_k\}_{k=1}^K$ appropriately to deal with the issue of different covariate sizes. Thus we do not need to treat these extra weights as tuning parameters.
\item We can also adjust the weights between TV and $\ell_1$ penalty by adjusting $\alpha \in [0,1]$. When $\alpha = 1$ TV regression reduces to Lasso.
\end{compactitem}

\subsection{Solving TV Regression}
\label{subsec:solve}
The additive model in \eqref{eqn:tvregreg} can be viewed as a linear transform on the model parameters $u_0$ and $u := [u_1; u_2; \dots; u_K]$:
\begin{equation}\label{eqn:additive2}
	\tau = u_0 \vct{1} + A u,
\end{equation}
in which $\vct{1} \in \mathbbm{R}^{N_x}$ is all one vector and $A \in \mathbbm{R}^{N_x\times |u|}$ is the corresponding linear operator. Here, $N_x = \Pi_{d=1}^D |V_d|$ is the number of all possible covariate combinations for $x$. TV in \eqref{eqn:tvregreg} can be rewritten as
\begin{equation}\label{eqn:TV2}
	TV(\tau) = \|D u\|_1 = \sum_{k=1}^K w_k \|D_k u_k\|_1,
\end{equation}
where $D_k$ is the (sparse) matrix which encodes the edges and $\ell_1$ penalty in~\eqref{eqn:tvfor1st} and \eqref{eqn:tvforhighorder}, and $D$ encodes all the information including the weights. With $M := \text{diag}\{M_e(x)\}$, the TV regularized regression~\eqref{eqn:tvregreg} can be rewritten as
\begin{equation}\label{eqn:tvregreg2}
\min_{u_0, u} \quad  \frac{1}{2} (u_0 \vct{1} + A u - \hat{\tau})^T M (u_0 \vct{1} + A u - \hat{\tau}) +  \lambda \|D u\|_1.\\
\end{equation}
The main problem is to solve $u$ and then the constant term $u_0$ follows trivially. Define $\mtx{P}_{\vct{1}} = \frac{\vct{1} \vct{1}^T M}{\vct{1}^T M \vct{1}}$, $B = A^T M (I - \mtx{P}_{\vct{1}})A$, $b = A^T M (I - \mtx{P}_{\vct{1}}) \hat{\tau}$ and $c = \hat{\tau} M (I - \mtx{P}_{\vct{1}}) \hat{\tau}$. \eqref{eqn:tvregreg2} can be decomposed as first solve
\begin{equation}\label{eqn:tvregreg3}
\min_{u} \quad  \frac{1}{2}(u^T B u - 2 b^T u + c) + \lambda \|D u\|_1,
\end{equation}
and then $u_0 = \frac{\vct{1}^T M (\hat{\tau} - A u)}{\vct{1}^T M \vct{1}}$.

We solve \eqref{eqn:tvregreg3} using an ADMM algorithm. Details are put in the Appendix.

\subsection{How to Choose Weights?}
\label{subsec:weights}
It is crucial to choose the correct weights $\{w_k\}_{k=1}^K$ in our total variation~\eqref{eqn:tvforadditive}. We provide a data-adaptive method which chooses weights based on the training data alone without a separate test set and expensive parameter tuning. This method of choosing weights can be also applied to other problems, such as group lasso \citep{yuan2006model} with/without overlap groups.

The basic idea is that the weights $\{w_k\}_{k=1}^K$ for corresponding additive terms $u_k$ should play the role of balancing the penalties such that they don't over-penalize any term because the number of edges is larger. To achieve that, we propose a thought experiment. If there is no HTE, i.e. $u = 0$ in \eqref{eqn:tvregreg2}, then as we decrease $\lambda$ from $\infty$ to 0, there exists a range of $\lambda$ that the solution of \eqref{eqn:tvregreg2} contains nonzero $u_k$. We know these are noises and the whole purpose of penalty terms is to suppress them. In the same line of thinking, if our choice of $\{w_k\}_{k=1}^K$ is ``fair'', then those noises should show up at the same $\lambda$ in the path. If $u_d$ keeps turning into nonzero before $u_f$, then we over-penalized $u_f$ compared to $u_d$. In other word, we choose weights such that when we decrease $\lambda$ all $u_d$ will join the solution path at the same time (on average).

\begin{theorem}\label{thm:tvregzero}
 The unique minimizer of \eqref{eqn:tvregreg3} is $u = 0$ if and only if
    \begin{equation}\label{eqn:tvregzero}
        \sup_{u \neq 0} \frac{u^T b}{\|D u\|_1} \le \lambda
    \end{equation}
Let $b = [b_1; b_2; \dots; b_K]$ and with $\|D u\|_1 = \sum_{k} w_k \|D_k u_k\|_1$, $u = 0$ if and only if
    \begin{equation}\label{eqn:tvregzero2}
        \max_{1\le k \le K}\sup_{u_k \neq 0} \frac{u_k^T b_k}{w_k \|D_k u_k\|_1} \le \lambda.
    \end{equation}
Let $k^*$ be the $k$ that $\sup_{u_k \neq 0} \frac{u_k^T b_k}{w_k \|D_k u_k\|_1} = \lambda$, $u_{k^*}$ joins the solution path at $\lambda$.
\end{theorem}
Theorem~\ref{thm:tvregzero} can be proved using Lasso dual, as in \citep{hastie2015statistical}. Let
\begin{equation}\label{eqn:dual}
    \gamma_k = \sup_{u_k \neq 0} \frac{u_k^T b_k}{\|D_k u_k\|_1},
\end{equation}
and left hand side of \eqref{eqn:tvregzero2} can be simplified as $\max_{1\le k \le K} \frac{\gamma_k}{w_k}$. Recall $b = A^T M (I - \mtx{P}_{\vct{1}}) \hat{\tau}$ is a random vector. We pick $\{w_k\}_{k=1}^K$ such that $\gamma_k/w_k$ are all the same in expectation when $\hat{\tau}$ is pure random noise:
\begin{equation}\label{eqn:weights}
	w_k \propto \operatorname{\mathbb{E}}(\gamma_k) \quad \forall k = 1,2,\dots, K,
\end{equation}
where $b_k$ is the k\textsuperscript{th} entry of the random vector $A^T M (I - \mtx{P}_{\vct{1}}) \vct{n}$ where $\vct{n}\sim \mathcal{N}(0, M^{-1})$.

\subsection{Parameter Tuning for $\lambda$}
\label{subsec:lambdatuning}
When $\lambda$ is sufficiently large, the solution of our TV regression is 0, as shown in Theorem~\ref{thm:tvregzero}. As $\lambda$ decreases, more and more heterogeneous treatment effects come out, and finally $u$ becomes the standard least square solution when $\lambda = 0$. In other words, the model overfits the data when $\lambda$ is too small, but suppress all signals when $\lambda$ is sufficiently large, see Theorem~\ref{thm:tvregzero}. It is important to stop at the right $\lambda$: the one that ideally captures the true heterogeneous treatment effects and suppress the false positives.

We achieve these two goals by simply solving Eqn.~\eqref{eqn:tvregreg3} with multiple $\lambda$'s in a descending order. This can be very efficient because we can use the solution for the last $\lambda$ as the initial parameter for the next $\lambda$ and the two solutions should be very close, see \citep{hastie2015statistical}. For $\lambda$ selection, to avoid computational expansive alternatives such as cross validation or a separate test set. We use information criteria such as AIC \citep{akaike1998information} and BIC \citep{schwarz1978estimating}. Since TV Regression, like Lasso, will introduce bias, for each $\lambda$, after clusters have been discovered by TV Regression, we fit an OLS (ordinary least square) using only useful additive terms without the TV penalty in~\eqref{eqn:tvregreg3}, and record its residual $Res(\lambda)$ (weighted by $M_e(x)$ at each $x$). This model re-fitting is similar to Lasso de-biasing and is a common practice. We can define AIC and BIC of the de-biased model as
\begin{align}
    BIC(\lambda) &= 2 Res(\lambda) + Dof(\lambda) \log(N_t).\label{eqn:bic}\\
    AIC(\lambda) &= 2 Res(\lambda) + 2 Dof(\lambda)\label{eqn:aic}
\end{align}
where $Dof(\lambda)$ is the degree of freedom of the linear model and equals 1 plus the number of additive terms in the linear model, $N_t$ is the number of points where $M_e(x) > 0$ (pixels in the ``image''). We pick the $\lambda$ (or the associated model) which has the smallest information score. As a side benefit from fitting the OLS, we can also perform significant test \citep{allofstat} on each effect terms to decide whether they might be false signals.

\subsection{Post TV Regression Reprocess}\label{subsec:posttv}
TV regression help us to identify block-wise clusters that can be used to concisely summarize the HTE $\tau(x)$. These identifies clusters can then be treated as building blocks and used as new features to reprocess the data so that might be able to further improve conciseness of the representation. To see that, suppose there are two clusters A and B for browser and there exist a global effect 0 and 0.1 on cluster A and 0 on B. When A and B have the same size, we can see a solution of 0.05 global effect, 0.05 for A and -0.05 for B has the same TV penalty as the ground truth. If we define concise as the smallest number of HTE effects we need to use to summarize the structure, then TV regression might fail to identify the most concise solution. Similarly when a ground truth second order effect covers a big portion of the population, TV regression might identify it as 2 first order effects and 1 second order effect (every second order effect can equivalently be represented by one first order effect subtracts another first order effect and adds one second order effect.). A post reprocess also means TV regression solution is allowed to contain some false positive effect terms as long as all true block structures are captured, giving us more flexibility and robustness so that $\lambda$ tuning does not need to be perfect.

Minimizing the number of effects used to describe HTE can be achieved by a linear model with $\ell_0$ penalty. We take all first and second order clusters and its complements identified in TV Regression. Note that complement of a second order block is not itself a block, we need to further break it into blocks. We fit \eqref{eqn:master} using these first order cluster and second order blocks as predictors with an ElasticNet penalty \citep{zou2005regularization,mazumder2012sparsenet} that closely approximates $\ell_0$ penalty. An additional step of fitting an OLS using only nonzero terms in the result of ElasticNet can be used to debias the estimated effects and also provide confidence intervals and significant tests.

Confidence intervals and statistical tests in the above OLS doesn't take into account the fact that the model is from careful model selection using TV Regression and ElasticNet. A more recent area of research aims to provide correct post-selection inference \citep{taylor2014post,taylor2015statistical} and also a more principled way to search best model in a Lasso-like solution path. Results there can also be applied to TV Regression since it belongs to the family of generalized Lasso.

\subsection{Multiplicative Effect}
\label{subsec:multiplicative}
It is very common in practice that the relative effect (percent change) ${y^{(1)}}/{y^{(0)}}-1$ is of interest, we can fit an multiplicative model for ${y^{(1)}}/{y^{(0)}}$, i.e.,
\begin{equation}\label{eqn:multiplicative}
	\frac{y^{(1)}}{y^{(0)}} = v_0 \Pi_{k=1}^K v_k(x).
\end{equation}
Taking log on both sides, we can apply our TV regression~\eqref{eqn:tvregreg} to $\log(y^{(1)}/y^{(0)})$, i.e.,
\begin{equation}\label{eqn:multiplicative2}
	\log\left(\frac{y^{(1)}}{y^{(0)}}\right) = u_0 + \sum_{k=1}^K u_k(x).
\end{equation}
Now the measurement at $x$ are $\log(\bar{y}^{(1)}(x)/\bar{y}^{(0)}(x))$ whose variance can be estimated by the Delta Method \citep{asympstat} as:
\begin{equation}\label{eqn:variance2}
	\hat{\operatorname{\mathrm{var}}}_{\log}(x) = \frac{\operatorname{\mathrm{var}}^{(0)}(x)}{(\bar{y}^{(0)})^2} + \frac{\operatorname{\mathrm{var}}^{(1)}(x)}{(\bar{y}^{(1)})^2}.
\end{equation}
The matrix $M = \text{diag}\{1/\hat{\operatorname{\mathrm{var}}}_{\log}(x)\}$ can be computed correspondingly.

\subsection{Signal Importance Score}\label{subsec:score}
We define importance of an additive term by comparing the prediction error with and without an extra term. Typically prediction error need to be evaluated on a hidden test set. We use Stein's unbiased estimator which does not require testing on a separate test set.

\subsection{Covariate Pre-Screening}\label{subsec:prescreen}
When the number of covariate is large, the number of covariate combinations can be larger than the original sample size N. Moreover, the optimization problem \eqref{eqn:master} involves $M_e(x)$ which relies on reasonable variance estimations of $var^{(w)}(x), w=0,1$. When sample sizes $N^{(w)}(x),w=0,1$ are small, variance estimation and effect estimation $\hat{\tau}(x)$ can be unstable. To overcome that, we can simply use a Group Lasso \citep{yuan2006model} step to first screen out irrelevant covariates. This pre-screening stage does not need to achieve effect clustering, which is the task for the following TV regression on a much smaller set of covariates.

\section{Empirical Results}
\label{sec:examples}
In this section we apply TV Regression to both synthetic data and also real experiment data. Throughout this section we visualize a solution of TV Regression using a 2d matrix heatmap. This visualization is only for technical illustration and is not the final concise summary. Ground truth solution for Example~\ref{ex:1} is shown in Figure~\ref{fig:synthetic_truth}.
Recall that the ground truth is global effect 0.03, first order -0.1 on $X_2$:$2$ and second order effect 0.1 on the block  $X_1$$:$$(4,5,6,7)$ $\cap$ $X_3$$:$$(3,4)$. We concatenate all levels of 4 covariates together as both x-axis and y-axis and form a matrix where
\begin{compactenum}
   \item Diagonal entries represents first order effect $u_d(x_d)$,
   \item Upper triangular entries represents second order effect $u_{d,f}(x_d, x_f)$,
   \item The whole lower triangular entries are the same, representing global effect $u_0$.
\end{compactenum}
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.45\textwidth]{synthetic_truth}
    \caption{Example~\ref{ex:1}: ground truth effects in a matrix heatmap. Covariate names for y-axis is omitted. }
    \label{fig:synthetic_truth}
\end{figure}
In Figure~\ref{fig:synthetic_truth} the lower triangular displays global effect 0.03. The white pixel on the diagonal line is the -0.1 effect for $X_2$:$2$ and the dark blue block in the upper triangular stands for the 0.1 second order effect.

\subsection{A synthetic example}
\label{subsec:synthetic}
We run TV-regression to fit a second-order model~\eqref{eqn:secondordermodel} for Example~\ref{ex:1} in Section~\ref{subsec:issue}. For TV penalty, we simply use a complete graph on all 4 covariates assuming they are all categorical without a natural topology. We take $\alpha = 0.5$ in Eqn.~\eqref{eqn:tvfor1st} and \eqref{eqn:tvforhighorder} to equally weight the total variation and $\ell_1$ penalty. We use 10,000 random samples to estimate the weights $w_k$ in Eqn.~\eqref{eqn:weights}. We visualize the solution path (after coefficient debias using OLS for each $\lambda$), as described in Section~\ref{subsec:lambdatuning}.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{synthetic_0_scale}
    \includegraphics[width=0.225\textwidth]{synthetic_1_scale}\\
    \includegraphics[width=0.225\textwidth]{synthetic_2_scale}
    \includegraphics[width=0.225\textwidth]{synthetic_3_scale}\\
    \includegraphics[width=0.225\textwidth]{synthetic_4_scale}
    \includegraphics[width=0.225\textwidth]{synthetic_ls_scale}
    \caption{Solution path ordered row-wise from left. The six plots corresponds to $1/\lambda$ values at $0.5$, $0.8$, $2.49$, $4.372$, $4.442$ and $+\infty$. See Table~\ref{tab:synthetic_bic}.}
    \label{fig:synthetic_path}
\end{figure}
The solution path start with only global effect on the top left, first detected first order effect on $X_2:2$ when $\lambda$ decreases. It then detect first order effect $X_3$$:$$(3,4)$ and $X_1$$:$$(4,5,6,7)$ (middle plots). It then figured out a better representation is a second order effect  $X_3$$:$$(3,4)\cap X_1$$:$$(4,5,6,7)$. We point out that the first order effects on X1 and X3 are still nonzero at this stage, but are too small to be visible in Figure~\ref{fig:synthetic_path} bottom left. As we further decrease $\lambda$ the solution converges to the least square solution.

We tabulate the BIC score for all the models above in Table~\ref{tab:synthetic_bic}. We select the model with $1/\lambda = 4.442$ (Figure~\ref{fig:synthetic_path} bottom left) since it has the smallest BIC score. We can see that the selected model exactly captures the two effects in the ground truth, even though it also contains two very small first order effect on $X_3$$:$$(3,4)$ and $X_1$$:$$(4,5,6,7)$. To further improve the result, we applied a post TV reprocess as described in Section~\ref{subsec:posttv}. The effect of this reprocess is shown in Table~\ref{tab:synthetic_reprecess}. Reprocess removed the two false small first order effects mentioned above, and report an equivalent alternative presentation for the ground truth: instead of reporting the first order effect of -0.1 on $X2$$:$$2$, it reports first order effect of 0.1 on $X2$$:$$(1,3)$, and reduce the global effect by 0.1 to -0.07. They are equally concise by the number of HTE terms.
\begin{table}[!htb]
\centering
\begin{tabular}{@{}llllllll@{}}
\toprule
$1/\lambda$ & 0.5  & 0.8  & 2.49 & 4.372 & 4.442 & 7   & 14  \\ \midrule
\# Effects  & 0    & 1    & 2    & 3     & 4     & 5   & 7   \\ \midrule
BIC         & 4283 & 2574 & 1492 & 1159  & 632   & 638 & 644 \\ \bottomrule
\end{tabular}
\caption{The solution path of Example~\ref{ex:1}}
\label{tab:synthetic_bic}
\end{table}

\begin{table*}[!tb]
\centering
\begin{tabular}{@{}lllllllll@{}}
\toprule
              & global & X1:(4,5,6,7) & X1:(1,2,3,8,9,10) & X2:2   & X2:(1,3) & X3:(3,4) & X3:(1,2,5) & X1:(4,5,6,7)$\cap$X3:(3,4) \\ \midrule
TV Regression+OLS & 0.027  & -0.0009*     &                   & -0.098 &          & 0.0026*  &            & 0.096                      \\
Elastic Net+OLS   & -0.072 &              &                   &        & 0.101    &          &            & 0.103                      \\ \bottomrule
\end{tabular}
\caption{Two weak first order effects(*) in TV Regression result are removed after the reprocess. They also do not pass t-test in the OLS fit (p-value 0.66 and 0.15). All other signals pass t-test in OLS with very small p-values. Reprocessing with Elastic Net found an equivalent representation with the same number of HTE term as the ground truth. }
\label{tab:synthetic_reprecess}
\end{table*}

\subsection{TV-regression vs. Lasso}
\label{subsec:comparelasso}
We also obtained the lasso path ($\alpha = 1$ in \eqref{eqn:tvforhighorder}) for Example~\ref{ex:1} and compared it to that of TV-regression with $\alpha = 0.5$. The BIC selected models are shown in Figure~\ref{fig:synthetic_comp}.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{synthetic_4}
    \includegraphics[width=0.225\textwidth]{synthetic_lasso}
    \caption{Comparison between TV-regression with $\alpha = 0.5$ (left) and Lasso (right, TV-regression with $\alpha = 1$). Both models are selected by BIC.}
    \label{fig:synthetic_comp}
\end{figure}
We saw that
\begin{compactenum}
\item Although Lasso captures both first and second order effects, it wasn't designed to discover the block structure of the second order effect.
\item Lasso also contains more false signals than TV Regression. We believe this is still true if cross-validation is used for $\lambda$ tuning, as reported in \citep{tian2014simple}.
\end{compactenum}
The true signal in Example~\ref{ex:1} is actually relatively strong comparing to the noise. If we lower the strength of signal from 0.1 and $\pm0.1$ to $\pm0.05$ and run the same example, as shown in Figure~\ref{fig:synthetic_comp_weak}, we found Lasso completely missed the second-order block with many false signals! In comparison, TV regression still captures the group truth, with only one very weak false signal on $X_3$$:$$(3,4)$ which is later removed in post reprocess.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{synthetic_tv_weak_scale}
    \includegraphics[width=0.225\textwidth]{synthetic_lasso_weak_scale}
    \caption{Comparison between TV-regression with $\alpha = 0.5$ (left) and Lasso (right, TV-regression with $\alpha = 1$) with ground truth effects reduce to $\pm0.05$ from $\pm0.1$.}
    \label{fig:synthetic_comp_weak}
\end{figure}

\subsection{An experiment on a personalized recommendation}
\label{subsec:rec}
\begin{ex}\label{ex:2}
Microsoft ran an online controlled experiment to evaluate a new gender based personalized recommendation algorithm. We have two covariates (Gender and ProductType) in this example. The mean and standard deviation ($1/\sqrt{M_e}$ in Eqn.~\eqref{def:M}) of the percent change for an important metric are shown in Figure~\ref{fig:rec_data}.
\end{ex}
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.48\textwidth]{rec_data}
    \caption{Example~\ref{ex:2}: mean and standard deviation of the percent change by gender and product type.}
    \label{fig:rec_data}
\end{figure}

We again use complete graph for both covariates since they are unordered. We run TV-regression with $\alpha=0.5$, and the first five models in the solution path and the least square solution ($1/\lambda=\infty$) are shown in Figure~\eqref{fig:rec_path}.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{rec_0_scale}
    \includegraphics[width=0.225\textwidth]{rec_1_scale}\\
    \includegraphics[width=0.225\textwidth]{rec_2_scale}
    \includegraphics[width=0.225\textwidth]{rec_3_scale}\\
    \includegraphics[width=0.225\textwidth]{rec_4_scale}
    \includegraphics[width=0.225\textwidth]{rec_ls_scale}
    \caption{Solution path of TV Regression for Example~\ref{ex:2}. Bottom left: $1/\lambda =  5.8$ (smallest BIC); bottom right: $1/\lambda = \infty$ (least square).}
    \label{fig:rec_path}
\end{figure}

We select the model with $1/\lambda = 5.8$ (Figure~\ref{fig:rec_path} bottom left) using BIC. The selected model contains two first order effects: strong Gender:Female effect and a weaker ProductType:Album\&Movie effect together with two second order effects: (Male, Album) and (Female, TV series). All the selected effects aligns well with the Figure~\ref{fig:rec_data} where there is certainly a first order effect for gender and female customers have more negative effect for Album and TV Series. The weaker first order ProductType:Album\&Movie effect has a p-value of 0.04 in the post reprocess stage and therefore is a boarder-line effect.

\subsection{Diagnosis in KPI Monitoring}\label{subsec:bing}
\begin{ex}\label{ex:3}
We observed the revenue per volume (RPV) of Product A for the recent week is lower than the week(s) before. We suspect this is not a systematic decline but rather a potential problem of a subpopulation. For illustration, we choose four covariates (DeviceOSName with 5 levels, DeviceType with 2 levels, ProductType with 13 levels and WeekDays with 7 levels). Real application was run on a large number of covariates so a pre-screening stage described in Section~\ref{subsec:prescreen} was used.
\end{ex}
The gross RPV can be calcuated from the RPV of subpopulations as below:
\begin{equation}\label{eqn:rpv}
    \text{RPV} = \frac{\sum_{i} \text{rpv}_{i} \text{vol}_i}{\sum_i \text{vol}_i} = \sum_{i} \text{rpv}_{i} \text{w}_i,
\end{equation}
where $w_k = \text{vol}_{i}/\sum_i \text{vol}_i$ is the weight of volumes from subpopulation $i$.

For covariates DeviceOSName, DeviceType and ProductType, we simply use complete graph to define the total variation. For WeekDays, we use the loopy graph in which two consecutive weekdays (Monday and Tuesday, Tuesday and Wednesday, ..., and Sunday to Monday) are connected. We compute the mean and standard deviation of the percent change treating pre-period as control and post-period as treatment for both $\text{rpv}_{i}$ and $w_i$, as in Example~\ref{ex:2}. We then run TV-regression with $\alpha=0.5$. Figure~\ref{fig:ex3} shows the BIC selected models.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{rpv}
    \includegraphics[width=0.225\textwidth]{weight}
    \caption{TV Regression results for percent changes of revenue per volume $\text{rpv}_i$ (left) and weights $w_i$ (right).}
    \label{fig:ex3}
\end{figure}

The results in Figure~\ref{fig:ex3} (left plot) suggested no HTE at all for revenue per volumne. For weights $w_i$ (Figure~\ref{fig:ex3} right), it showed a strong increase (40\%) on the first level of DeviceOSName. In fact, the first level of DeviceOSName was Android and it turned out that the volume from one device with Android system increased a lot in the post-period. Also, comparing to devices with other operation systems, Android devices had a relatively low rpv. Therefore, the volume increase in this low-rpv Android device caused the drop of the gross RPV.

\subsection{Adaptive Between-Graph TV Weights}
\label{subsec:weightexample}
We demonstrate the necessity of adaptive between-graph weights $w_k$ in the definition of total variation~\eqref{eqn:tvforadditive} using a synthetic example. We described our procedure in Section~\ref{subsec:weights}.
\begin{ex}\label{ex:4}
There are 3 categorical covariates $X_1,X_2, X_3$. $X_1$ is an ordered covariate with 20 equally weighted levels. Its graph connects every consecutive pair. $X_2$ has 10 equally weighted levels. $X_3$ has 5 levels, with proportion 10\%, 30\%, 30\%, 20\% and 10\%. Both $X_2$ and $X_3$ have a complete graph. We put a simple ground truth effect as this: $X_2$$:$$1$ has a first order effect of 0.015 and there's no other heterogeneous effects. In addition, there is a constant background effect -0.01. $10,000$ \textit{i.i.d.} treatment and control observations are simulated from $N(0, 0.1)$ respectively with treatment effect added to treatment observations according to the prescribed design.
\end{ex}
TV-regression with $\alpha = 0.5$ and a first order additive model is applied to this example. First we simply use equal weights, i.e. $w_1 = w_2 = w_3 = 1$ in~\eqref{eqn:tvforadditive}, part of the obtained solution path is shown in Figure~\ref{fig:ex4_noweights}. The BIC picks the model on the right in Figure~\ref{fig:ex4_noweights}. Note that $X_2$ and $X_3$ are categorical so in Figure~\ref{fig:ex4_noweights} only the lines connecting $X_1$'s levels mean ordering.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{plain_3}
    \includegraphics[width=0.225\textwidth]{plain_4}
    \caption{Part of the solution path with equal weights. Left: $1/\lambda =  13.4$; Right: $1/\lambda =  18.4$. BIC chose the model on the right.}
    \label{fig:ex4_noweights}
\end{figure}

Although the selected model contains the ground truth effect on $X_2$:$1$, there are many false positive effects due to noise. The reason is because $X_1$ is ordered, and there is considerably less edges in its TV graph compared to $X_2$ and $X_3$. If we use equal weights between the graphs, then false positive in $X_1$ will be less penalized than those in $X_2$ and $X_3$.
In comparison, we use Monte Carlo method with 10,000 samples to estimate the weights in Eqn.~\eqref{eqn:weights}. The results are
\begin{equation*}
    (w_1, w_2, w_3) = (88.5, 15.4, 32.5).
\end{equation*}
As expected, we put higher weights on $X_1$ and less on $X_2$ and $X_3$. $X_3$ got higher weights than $X_2$ because is only has 5 levels compared to $X_2$ having 10. The corresponding solution path is shown in Figure~\ref{fig:ex4_weights}. The BIC picks the model on the right side in Figure~\ref{fig:ex4_weights}.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.225\textwidth]{weight_1}
    \includegraphics[width=0.225\textwidth]{weight_2}
    \caption{Solution path with weights~\eqref{eqn:weights}. Left: $1/\lambda = 9.40$; right: $1/\lambda = 25.9$.}
    \label{fig:ex4_weights}
\end{figure}
We can see that
\begin{compactenum}
\item The selected model contains the true positive effect on $X_2$:$1$ and one false positive effect on $X_2$:$(2,5,7)$.
\item The ground truth effect on $X_2$:$1$ enters the solution path in the first place and has higher importance score than $X_2$:$(2,5,7)$.
\end{compactenum}
Therefore, TV-regression with weights~\eqref{eqn:weights} serves our goals better than TV-regression with equal weights. We also ran post reprocess. In this example, we weren't able to remove the false positive, which has a p-value of 0.001.

\section{Conclusion and Future works}\label{sec:con}
In this paper we presented Total-Variation regularized regression to achieve simultaneous heterogeneous regression and concise summarization. We bet on sparsity of effects and use an additive model so that HTE can be summarized using a small number of lower order effects. At the same time Total-Variation penalty will further cluster similar effects into block structures, which can be easily interpreted and are memorable as a set of AND clause. We described how to define Total-Variation penalty for both ordered and unordered discrete covariates in detail and proposed a novel method of tuning between-graph weights. 4 examples using both synthetic and real experiment data were used to demonstrate TV regression and compare it to existed approaches such as Lasso.

One line of future work is to use more rigorous post selection inference to assess the strength of discovered effects and further remove false positives. We had some success using post-TV reprocess, but the t-test we used ignored the fact that the model is selected and therefore still produces more false positive than prescribed significant level.

Another line of future work is to compare TV regression to alternative fusion penalties such as OSCAR \citep{bondell2008simultaneous}, GOSCAR as well as other non-convex penalties \citep{yang2012feature}.
\setlength{\bibsep}{0.0pt}
\bibliographystyle{agsm_nourl}

redpajama_set_name: RedPajamaArXiv


The Ministry of Energy and Minerals is responsible for the national energy policy and coordination of the activities in the sector. Its task is to create an attractive framework so that all stakeholders can operate successfully. MEM has responsibility for creating and reviewing policies and strategies as set out in the National Energy Policy and the National Energy Strategy through issuances and guidelines.
The Energy and Water Utilities Regulatory Authority is an autonomous multi-sectoral regulatory authority established by the Energy and Water Utilities Regulatory Authority Act in 2001. It is responsible for technical and economic regulation of the electricity, petroleum, natural gas and water sectors.
The functions of EWURA include licensing, tariff review, monitoring performance and standard-setting with regards to quality, safety, health and environment. EWURA is also responsible for promoting effective competition and economic efficiency, protecting the interests of consumers and promoting the availability of regulated services to all consumers including low income, rural and disadvantaged consumers in the regulated sectors.
Tanzania Electric Supply Company Limited is a public organization under the Ministry of Energy and Minerals. The company generates, transmits, distributes and sells electricity to the Tanzania Mainland and sells bulk power to the Zanzibar Electricity Corporation (ZECO). TANESCO in the year ending June 2016 was responsible for 60.72% of the electricity generated in the country[i], also being responsible for all the centralized transmitting and distributing facilities in Tanzania Mainland. The Government of Tanzania removed TANESCO's monopoly as the sole power generating and distributing company in 2008 after setting first steps towards market liberalization in 1992. The Company is currently facing significant challenges underpinned by issues on its financial performance.
REA is an autonomous body under the Ministry of Energy and Minerals. Its main role is to promote and facilitate improved access to modern energy services in rural areas. Deriving its power from the Rural Energy Act no.8 of 2005, the agency became operational in October 2007. The agency actively works with the private sector, non-governmental organizations, community-based organizations and government agencies.
The Tanzania Investment Centre (TIC) was established in 1997 by the Tanzania Investment Act to be the Primary Agency of the Government to coordinate, encourage, promote and facilitate investment in Tanzania and to advise the Government on investment policy and related matters. The agency deals with all enterprises whose minimum capital investment is not less than US $ 500,000 if foreign owned or US $ 100,000 if locally owned. The agency assists all investors to obtain permits, authorization etc. required by other laws to set up and operate investments in Tanzania.
The Tanzania Revenue Authority (TRA) was established by Act of Parliament No. 11 of 1995, and started its operations on 1st July 1996. In carrying out its statutory functions , TRA is regulated by law, and is responsible for administering impartially various taxes of the Central Government.
TIB Development Bank Limited was established in November 1970 under a Parliamentary Act. The Government re-designated TIB as a Development Finance Institution (DFI) in 2005 with engagement now focused primarily on infrastructure, industrialization (agro-processing, mining, and general manufacturing) oil and gas and services sector.
The Tanzanian government differentiates between Independent Power Producers, Small Power Producers and Emergency Power Producers. They currently contribute 40% to the installed capacity. Nine large power plants using gas, heavy fuel oil and diesel are run by private companies (IPPs). Two Emergency Power Producers, Aggreko and Symbion LLC, were hired by TANESCO to temporarily produce power in order to decrease the need for load shedding. Independent Power Producers are all private companies running power plants larger than 10 MW under a Purchase Power Agreement. A number of Small Power Producers (SPPs) develop smaller projects (< 10 MW) under a specific SPP framework and sell their produced electricity to TANESCO or to consumers such as agricultural businesses that produce sugar, tea, sisal or tannin. Their supply is usually based on renewable energy technologies – particularly biogas and solar PV.
The development vision aspires for Tanzania to have been transformed into a middle income and semi industrialized nation by 2025, characterized by (i) high quality and sustainable livelihoods; (ii) peace, stability and unity; (iii) good governance and the rule of law; (iv) an educated and learning society; and (v) a strong and competitive economy.
The National Strategy for Growth and Reduction of Poverty (NSGRP II or MKUKUTA II) is an organizing framework to rally national efforts in accelerating poverty-reducing growth. It is a second in series of results-based national strategies. Like its predecessor – MKUKUTA I, the strategy puts emphasis on the role of governance and growth for poverty reduction. The Strategy is divided into three main clusters, cluster one on Growth and Reduction of Income Poverty with affordable and reliable energy as one key sub area.
The Second Five Year Development Plan (FYDP II) has integrated frameworks of the merged FYDP I (2011/2012-2015/2016) and the NSGRP/MKUKUTA II, with a designated theme of nurturing an Industrial Economy. Energy sector challenges are highlighted as low access, and high costs of power production, distribution and transmission. A situation outlined as being caused by a 40-year Government monopoly that has stifled development in the sector, an overreliance on hydropower and inefficiencies of the power utility, TANESCO. Swift reform is placed at the centre of development in this critical sector under the FYDP II, including a separation of powers in the following key functions: determining power purchasing tariffs, management of the power master plan, and power transmission.
The main elements of the policy address the development of efficient domestic energy resources, ensuring market-determining energy prices, improving the reliability and security of electricity as well as commercializing and privatizing the energy sector.
The Act establishes a Rural Electrification Board, Agency and Fund and describes in considerable detail the functions and relationships of these different entities. The Tanzanian law also provides more detail on sources of financing for the fund. The Act is intended to promote improved access to modern energy services in rural areas of mainland Tanzania and providing grants to developers of rural energy projects and to TANESCO for rural grid distribution investments.
The Act liberalised the production and distribution of electricity in Tanzania, which was previously reserved for TANESCO. The Act provides for the facilitation and regulation of the generation, transmission, transformation, distribution, supply and use of electric energy, cross-border trade in electricity, and the planning and regulation of rural electricity in Tanzania.
Tanzania introduced a feed-in tariff scheme for small power producers (100 kW to 10 MW) in 2008 with the framework being reviewed in 2015. In the Second Generation SPP (Small Power Producer) framework, EWURA applies two approaches depending on the technology: Renewable Energy Feed-In Tariffs are applied for small hydro and biomass projects whereas a bidding approach is applied for solar and wind projects. The Second Generation SPP framework is based on guiding principles that include SPPs receiving a fixed tariff for the duration of the Small Power Purchase Agreement (SPPA).
In August 2015, the Ministry issued Model Power Purchase Agreements for seven energy technologies to be used as guidance for projects larger than 10 MW. These include the technologies hydro, natural gas, oil, coal, geothermal, solar and wind. The Model PPAs were issued for the use in negotiations between project developers and the power off-taker.
Short and medium term projects have been identified in the Ministry's Power System Master Plan 2007-2035 to replace short-term emergency generation plants. Proposed projects include the processing of a number of PPAs relating to interconnections with neighboring countries and IPPs, the construction of new generation facilities (including 100 MW at Ubungo, 60MW at Nyakato – Mwanza, and 222 MW at the Rumakali Hydropower Project), the expansion of the high-voltage transmission network and further studies into wind energy projects. In total the PSMP, as updated in 2012, foresees the development of 3,564 MW of renewable energy capacity from hydro, wind, solar and biomass plants.
The Government of Tanzania (GOT) has historically been successful in attracting FDI, with recent years being focused on the productive and extractive sectors. The aforementioned Tanzania Investment Center (TIC), sits at the heart of the efforts to attract FDI through promoting and facilitating investments in the country. TIC approved projects along with those in Zanzibar through the Zanzibar Investment Promotion Authority (ZIPA) receive certificates of incentives which include VAT and import duty exemptions and 100% repatriation of profits, dividends, and capital after tax. Sectors of focus under the investment institutions include energy in addition to amongst others agriculture, tourism and transportation. Registering with the TIC offers incentives for joint ventures with Tanzanian and wholly owned foreign projects investing a minimum of $500,000.
In 2014 the amended version of the Public-Private Partnership (PPP) Policy with the PPP Act and PPP Regulations came into law, covering many areas of foreign investment including power generation and transmission.
The Tanzanian government established a Local Content Policy (LCP), originally for the oil and gas industry, in May 2015. This was then expanded to cover all economic sectors including in energy with an objective of placing local products, produced by local industries/organization and businesses in an enhanced position for accessing opportunities from foreign direct investments.
The Rural Energy Fund (REF) is an instrument administered by the Rural Energy Agency (REA) to promote programmes designed to increase energy access in rural Tanzania by subsidizing rural energy projects via grants towards capital costs of project implemented by public and private organisation as well as cooperatives and community groups. The fund is capitalised through government budget allocations, interests or return on investment, specific taxes on petroleum products and electricity, and development agencies. One specific example under this framework is the Result Based Finance (RBF) grants programme for mini and micro-grids as supported by SIDA and DFID that had its first call for proposals at the end of 2016.
Numerous development agencies and partners active in Tanzania cover energy within their areas of focus. These include but are not limited to the EU, UK Department for International Development (DFID), Swedish International Development Cooperation Agency (SIDA), International Development Association (IDA) and Norwegian Agency for Development Cooperation (NORAD).
IDA amongst other initiatives has supported developments through the Rural Energy Agency and the Ministry of Water on using Solar Energy to power water supply in the country. Current plans aspire for all new rural water projects in Tanzania to utilise solar power. SIDA aims to directly support access to electricity for 300,000 Tanzanians with sustainable energy promotion through an improved regulatory framework and private sector engagement being the focus. One such example, as highlighted above, being the green mini-grid RBF programme jointly developed with DFID. NORAD reports investing over EUR 9.5million in Tanzania in 2015 in the environment and energy sectors. Over the years this work has included supporting the supply of power to Zanzibar through an undersea cable and additional follow up support to the Zanzibar Electricity Company (ZECO). Further support has included funding pilot studies for a regional power line between Tanzania and neighbouring Kenya and on grid development plans with TANESCO. More recently Norway has funded the drawing up of a now adopted 10-year investment plan for village electrification that could see the country further fund grid expansion and private investment in isolated networks, as well as to distributed systems in partnership with Lighting Africa Tanzania.