text:
Practice areas Jurisdictions Compare & Research Business Sectors Latest News GLG Events Videos
Free Newsletter View Basket Search
Arbitration & ADR
Competition & Business Crime
Third-Party Finance
People & Firms
ICLG.com > Commercial Dispute Resolution > Feris looks to arbitration's pastures new
Feris looks to arbitration's pastures new
Published by: Andrew Mizner at 17/08/2017
Growth in Latin American international arbitration is on the mind of former ICC Court deputy secretary ge...
Growth in Latin American international arbitration is on the mind of former ICC Court deputy secretary general José Ricardo Feris, as he joins Squire Patton Boggs and reflects on his role in the ICC's expedited procedure rules.
"Once I had mastered the type of work that I was doing, it was time for me to do something different and move to the other side of the table," says José Ricardo Feris, explaining his move to Squire Patton Boggs in Paris as a dispute resolution partner.
For Feris it was time for a change after 13 years spent with the International Chamber of Commerce International Court of Arbitration (ICC), during which he rose from the rank of deputy counsel to spend his final five-and-half years as deputy secretary general.
He made his first foray into private practice on 18 July, starting work as an international dispute resolution partner with Squire, a move that was announced in December.
It is a well-trodden path, with many of his former colleagues either having made the same move, or having begun their careers in private practice before joining the ICC.
Feris cites Squire's track record in both commercial and investment arbitration, the growth of the practice and the firm's desire to grow it in Paris, the city in which he lives and wished to stay, as reasons for choosing the firm.
LATAM LOVE
Another draw for Feris was the firm's ambitions in his "natural market" of Latin America. The firm has an office in Miami, which is a hub for work in the region – "a place that many Latin Americans look up to" and an office in the Dominican Republic, where Feris, himself Dominican, knows the team.
Arbitration is on the rise in Latin America and the ICC itself has made the region a priority. In 2015, at a conference in Miami, the ICC announced its intention to open an office somewhere on the South American continent and host an event in Cuba.
Two years later, both have come to pass, siting a case management team in São Paulo, Brazil, in May this year and two conferences in two years in Havana, with another event planned next month, this time in Lima, Peru.
"The international arbitration market in Latin America has become a very sophisticated one," says Feris. "You now have a lot of arbitration practitioners that have a lot of experience and are involved in very complex and very sophisticated arbitration proceedings. You have governments that are, to a great extent, supporting arbitration [through] either the enacting of arbitration legislation or even reforming arbitration laws."
He points to Peru, where arbitration is mandatory and the government has consistently updated its Arbitration Act to keep it up to date, "much more than any other country in the world", Brazil, which has introduced specific laws to regulate arbitration with the state and state agencies, particularly when it comes or public-private partnership", and Mexico, which has created a framework for arbitration in the energy sector.
"So the elements are there for arbitration to continue growing in Latin America. Judges are becoming more expert when it comes to the application and the understanding of the basic arbitration legal theories and mortal provisions."
As someone who so recently worked at the ICC, Feris says that "its challenge is to compete globally with institutions which actually compete locally or regionally."
He categorises competitors such as the London Court of International Arbitration in Europe, the Singapore International Arbitration Centre in Asia and the International Centre for Dispute Resolution in the Americas as regional institutions. They might quibble with that definition and argue that they are genuinely international, but Feris insists that "the ICC is trying to solidify its status as the only truly global arbitration institution".
He points to the announcement in June that it will open a second case management office, in Singapore and believes that the Brazilian office could expand to have a wider remit within Latin America.
"The second challenge for the ICC is to maintain the same level of quality throughout the offices that traditionally it was offering out of Paris," he adds.
GROWTH SECTORS
One of the initiatives Feris worked on during his tenure at the ICC was making it more attractive for investment arbitration, and this is an area he intends to practice in at Squire, alongside commercial arbitration, particularly in the construction sector, and mediation.
"Clients are much more demanding these days in terms of what the counsel can do for them. Not just getting them into the arbitration case or into litigation, but also trying to assist them with alternative dispute resolution mechanisms, and mediation is one of them."
Construction is a traditionally strong source of arbitral work and Feris expects to see "continuous growth in construction, particularly in the emerging markets" wherever there is economic growth, particularly in developing parts of the world, likewise in oil and gas.
One area where he sees more scope for growth is in finance, which has been historically resistant to alternative dispute resolution, a position which has been changing over the past few years:
"The 2008 crisis made state courts a little bit less attractive for banks and they are now involved in projects around the world where going to state courts is pretty much not an option. That is making banks reconsider their traditional position and consider international arbitration. If you look at the ICC statistics, you see an interesting peak in the number of arbitrations related to the banking and finance industry."
NEW RULES A SUCCESS
One of his final acts at the ICC was his involvement in the 1 March introduction of the expedited procedure provisions, which allow for more streamlined proceedings in smaller arbitrations. This was a crucial addition, as expedited proceedings have been in demand from parties and any institution hoping to keep up with competition from other institutions, other forms of alternative dispute resolution and the courts needed to respond.
"Looking at it now from a more objective angle I still think that [the rules] are going to be very successful," says Feris. "[Parties] don't want to spend too much time in arbitration."
This is reflected in the arbitration agreements being drawn up by companies, he says, which often specify a smaller, local arbitral institution, with only one arbitrator, for smaller disputes, reserving three-arbitrator panels at large international institutions for the larger cases only.
"These arbitration agreements created a lot of problems, because there was a lot of uncertainty as to what was going to happen," he says, so the introduction of standardised rules should meet the demand in a more consistent way.
While it will take time for parties and the ICC to become familiar with the operation of these procedures, as they did with the emergency arbitrator provisions introduced in 2012, he is confident that they will be a success. Now, however, he has moved to the other side of the table and will be advising clients on how best to use them.
ANOTHER NEW FACE
At the start of August, the firm's Denver office was joined by David Blake, former chief deputy attorney general for the state of Colorado, as a partner. Blake has brought his first-hand experience to the firm's state attorneys general practice group, advising the firm and its clients on the activities of those state officials, including litigation and prosecution.
The firm recruited US bankruptcy and insurance litigators in March, and in April it hired two Washington, DC, telecommunications disputes partners, one of whom, Eduardo Guzmán, is another with a strong Latin American practice.
CDR Spring 2019
Subscribe to CDR
Subscribe now and access all of CDR's daily news and analysis, complete with a full archive of all articles and PDF back issues of CDR magazine.
Holland & Knight makes up new partners
A group of 18 litigators has taken a step up as part of the Florida-founded firm's 37 partner promotions.
Robert Li | 21 January, 2021
Elevations at WilmerHale
Of the seven partner promotions at WilmerHale, two have dispute resolution practices.
US disputes moves and promotions
Latham & Watkins and McGuireWoods have made new hires, while Steptoe & Johnson has rewarded its up-and-coming litigators with partnerships.
Andrew Mizner | 20 January, 2021
Lateral Moves
Holland & Knight makes up new partners 21/01/2021
Elevations at WilmerHale 21/01/2021
US disputes moves and promotions 20/01/2021
Harneys buttresses BVI office 20/01/2021
BVI litigator reverts to Carey Olsen 20/01/2021
Six step up at Boies Schiller Flexner 20/01/2021
January hires in Belgium and Italy 19/01/2021
Multiple elevations for Debevoise 19/01/2021
CDR Magazine published by: Global Legal Group


a.	What agencies regulate securities markets?
b.	How are start-up firms usually financed?
c.	Differentiate between a private placement and a public offering.
d.	Why would a company consider going public? What are some advantages and disadvantages?
e.	What are the steps of an initial public offering?
f.	What criteria are important in choosing an investment banker?
g. Would companies going public use a negotiated deal or a competitive bid?
h.	Would the sale be on an underwritten or best efforts basis?
i.	Without actually doing any calculations, describe how the preliminary offering range for the price of an IPO would be determined?
j.	What is a roadshow? What is bookbuilding?
k.	Describe the typical first-day returns of an IPO and the long-term returns to IPO investors.
l.	What are the direct and indirect costs of an IPO?
m.	What are equity carve-outs?
n.	In what other ways are investment banks involved in issuing securities?
o.	What is meant by going private? What are some advantages and disadvantages?
p.	How do companies manage the maturity structure of their debt?
q.	Under what conditions would a firm exercise a bond's call provision?
r.	Explain how firms manage the risk structure of their debt with: (1) project financing, and (2) securitization.
Answer:	The main agency that regulates the securities market is the Securities And Exchange Commission. Some of the responsibilities of the SEC include: regulation of all national stock exchanges--companies whose securities are listed on an exchange must file annual reports with the SEC; prohibiting manipulation by pools or wash sales; controls over trading by corporate insiders; and control over the proxy statement and how it is used to solicit votes.
The Federal Reserve Board controls flow of credit into security transactions through margin requirements. States also have some control over the issuance of new securities within their boundaries. The securities industry itself realizes the importance of stable markets, therefore, the various exchanges work closely with the sec to police transactions and to maintain the integrity and credibility of the system.
Answer:	The first financing comes from the founders. The first external financing comes from angels, who are wealthy individuals. The next external financing comes from a venture capital fund. The fund raises capital from institutional investors, usually around $70 to $80 million. The managers of the fund are called venture capitalists. The fund invests in ten to twelve companies, and the venture capitalist sits on their boards.
Answer:	In a private placement stock is sold directly to one or a small group of investors rather than being distributed to the public at large. A private placement has the advantage of lower flotation costs; however, since the stock would be bought by a small number of outsiders, it would not be actively traded, and a liquid market would not exist. Further, since it would not have gone through the SEC registration process, the holders would be unable to sell it except to a restricted set of "sophisticated" investors. Further, it might be difficult to find investors willing to invest large sums in the company and yet be minority stockholders. Thus, many of the advantages listed above would not be obtained. For these reasons, a public placement makes more sense in Randy's situation.
·	Going public will allow the family members to diversify their assets and reduce the riskiness of their personal portfolios.
·	It will increase the liquidity of the firm's stock, allowing the family stockholders to sell some stock if they need to raise cash.
·	It will make it easier for the firm to raise funds. The firm would have a difficult time trying to sell stock privately to an investor who was not a family member. Outside investors would be more willing to purchase the stock of a publicly held corporation which must file financial reports with the sec.
·	Going public will establish a value for the firm.
·	The firm will have to file financial reports with the SEC and perhaps with state officials. There is a cost involved in preparing these reports.
The solution provides answers on various subjects from choosing an investment banker to bookbuilding to equity carve-outs for this mini case question. 2287 words.
Randy's Restaurant: raise $15M in new capital. Discuss equity market options.


text:
My Town: Beth Daniel
By Ron Sirak
Photo courtesy of the Charleston Place Hotel.
A large part of the charm of Charleston, S.C., is that while it was the cultural and economic center of the pre-Civil War South, economic malaise for much of the 20th century spared it the ravages of urban renewal. So much of the old city is perfectly preserved, it truly feels like a trip back in time. The porches and window boxes alone are worth the trip.
Beth Daniel was raised in the midst of the city's throwback joys. The LPGA Hall of Famer started playing golf at CC of Charleston when she was 6, and though she now lives in Florida, she returns often to visit family and friends. Needless to say her passion for her hometown has never moved away.
I like the Fazio course at Wild Dunes Resort. You really get a feel for the ocean and the beach. It's a neat layout that winds through the sand dunes. All the courses at the Kiawah Island Golf Resort are really good, but I play the Nicklaus one the most. The Ocean Course is great, but it beats you to a pulp. Of course, people like to play that one because they want to see it.
The GC at Briar's Creek, designed by Rees Jones, also is a lot of fun. You look across the river right into Kiawah, which is beautiful. It's private so you'll have to finesse your way on. Then there is Cassique GC, which is very unusual. Two other private courses that are worth making the effort to get on if you can arrange it with a member are the CC of Charleston, where I learned the game, and Yeamans Hall CC.
The Planter's Inn right on the corner of Market and Meeting Streets puts you in the middle of everything. The Charleston Place Hotel (Above), which is also on Meeting Street, is where my Hall of Fame party was. The rooms are awesome, and they have a great spa. The bed-and-breakfast places are a dime a dozen in Charleston, and they are all really good, very quaint and romantic.
Take a bus tour or a carriage tour and you can see all kinds of history. Just remember, some of the drivers make up their own history. They also have bike and walking tours of the downtown area. Go to Middleton Place Gardens or Magnolia Plantation and Gardens. The South Carolina Aquarium on the harbor is fantastic. The Confederate Museum has a lot of cool artifacts, and the Hunley Museum displays a submarine that was sunk in the harbor during the Civil War and was salvaged several years ago.

redpajama_set_name: RedPajamaCommonCrawl


In the progress to do this project, I understand that microeconomics is the analysis of individual economic products where we will make use of it in our daily lives. We will understand how to meet the company goals with limited resources with the better knowledge of the economic principles and theories. Besides that, we can also learn basic economics and also have a better understanding of the economics of the marketplace place. Through this report, I can understand more about the monopoly market, how it operates and its characteristics from my research. Out of this research, I've a good understanding on the monopoly on the market composition. Besides that, this research also trained me to differentiate the four market framework in terms of its characteristics which can be totally different in one and another.
According to Hashim, A. (2001) Comprehensive Economics Guide. 2nd ed. Singapore: Oxford College or university Press Pte Ltd, p. 84, Monopoly is an industry composed of a single seller of something with no substitutes and with high barries to entry. A monopoly electricity exists when a single firm control buttons more than 25% of a market.
3. 1. 1 A monopoly exists when there is only one seller of something. For instance, The Tenaga Nasional Berhad(TNB) has a monopoly of the electricity way to obtain Peninsular Malaysia. All houses and outlets who get resource from Tenaga Nasional Berhad(TNB) will need to pay their electricity costs.
3. 2. 1 The merchandise sold by way of a monopolist should be no close substitutes. A couple of no other electricity distributor in Malaysia. The only dealer is the Tenaga Nasional Berhad (TNB). There is no competition because of their product.
3. 3. 1 In a very perfect competition, you will see no single firm can influence the price and this is called price taker. The Tenaga Nasional Berhad(TNB) will have the energy to decide and control the purchase price in the market since there are no competition around.
3. 4. 1 In a monopoly market, you will see tight barriers to the admittance of new companies and the barriers of admittance are natural. A monopolist faces no competition because of barriers to admittance.
AAdvertising in a monopoly market depends on the merchandise sold. If the merchandise are luxury goods such as imported items then the monopoly will require some advertisement to promote the consumers on the goods. Local public tool such as the electricity by Tenaga Nasional Berhad (TNB) need no advertisement since the consumers know from where you can obtain such goods and they're the only company who provides electricity.
Monopolies can maintain super-normal revenue in the long run. As with all firms, earnings are maximised when MC = MR. Generally, the amount of profit depends upon the degree of competition in the market, which for a real monopoly is zero. At revenue maximisation, MC = MR, and result is Q and price P. Given that price (AR) is above ATC at Q, supernormal earnings are possible (area PABC).
With no close substitutes, the monopolist can derive super-normal earnings, area PABC.
A monopolist without substitutes would be able to derive the greatest monopoly vitality.
4. 1. 1 The perfect competition has large number of buyers and retailers. Organizations are price taker because the number of a single vendor sells in market is so small compared to the overall industry. Besides that, the price is always constant where the vendor can only make a decision the quantity to be sold rather than the price tag on selling of something. An example of the perfect competition is the duck providers. The price tag on the duck on the market is still is determined by the demand and offer. The sellers can never control the price of the duck on the market even if they have high production, it will not affect much for the reason that industry.
4. 1. 2 In a very monopolistic competition, there are a sizable number of vendors. The amount of firms are present in a monopolistic competition market is very poor competition. Due to the size of each organization which is small and hence, no individual firm can influence or control the marketplace price. Therefore, each organization follows an independent price-output coverage. The company that produces toothpaste is in the monopolistic competition where there are extensive brands of toothpaste in Malaysia such as Darlie, Colgate and Polleney. They are able to never affect or control the price in of their products on the market.
4. 1. 3 In an oligopoly, the amount of organizations is small but the size of the businesses is large. The market share of each organization is large enough to dominate the marketplace. A few organizations control the entire industry of oligopoly. Including the petroleum companies specifically Shell, BHP, Caltex. They are really large firms who've market stocks which able to dominate the market.
4. 1. 4 Under monopoly, there is merely one owner of something and large number of buyers are present. A monopolist is a cost manufacturer since there is only one seller no competition and it has the power to control the purchase price in the market. One of the types of a monopoly is the Tenaga Nasional Berhad (TNB) where their company products electricity for the whole Peninsular Malaysia.
4. 2. 1 There exists unrestricted flexibility of accessibility and exit of the organizations from the industry in the perfect competition and the monopolistic competition. A company can easily enter into the market and exit the marketplace anytime they would like to. No restriction is imposed. If any firm who wish to open a fruit farms and operate the business if he/she has the necessary factors of development ( land, labour and capital) he/she can always starts off the business even there's a lot of berry farms are present.
4. 2. 2 In an oligopoly market, there are many barriers to entrance. Although similar to a monopoly, the firms in an oligopoly will limit new firms to enter the market. The types of barriers to entrance are economies of scale, forces to combine, ownership of patents and copyrights to mention a few. This is illustrated briefly by the petroleum industry in Malaysia where Mobil, Shell, Petronas and Caltex which already can be found in the market plus they control the marketplace. The chances for a fresh firm to be produced in the petroleum industry in Malaysia is very low due to the huge capital investment that they need to have a posture in the market.
4. 2. 3 Beneath the monopoly market, there will be restricted independence of entry and there are legal restrictions that restrict the accessibility of new companies into the industry. Hence there will be no competition and competition for businesses who are in the monopoly market. Telekom Malaysia (TM) is a good exemplory case of a monopoly since there is only one home cell phone service in Malaysia which is Telekom Malaysia (TM) rather than any other firms.
4. 3. 1 The businesses in a perfect competition must sell homogenous products. Inside the perfect competition structure, buyers cannot differentiate products in conditions of quality, packaging, colour and design being that they are indistinguishable. Furthermore, the organization cannot charge another price for the same product which can be found on the market. A classic example of this is the telecommunication service agency in Malaysia which can be Digi, Maxis and Celcom. They offer customers with the same product on the market but buyers cannot differentiate their products no matter how, since they are yet.
4. 3. 2 The monopolistic competition market offers differentiated products which are not identical. Each companies will have their own method to differentiate their products from other vendors to obtain additional customers or consumers. Their products can be different in conditions of the look, advertising, branding, and labeling. For instance, when a perfume is effectively packaged in a field and called 'best perfume' then the product is monopolistic competition.
4. 3. 3 Products in the oligopoly may be differentiated or undifferentiated. In Malaysia, the exemplory case of oligopoly market are petroleum and automobiles where petroleum is identical while automobiles are differentiated products.
4. 3. 4 Under monopoly market, the products produced has no close substitutes or unique. Tenaga Nasional Berhad (TNB) is one of the example of monopoly who's the electricity supplier from local people utility without any close substitutes but if the buyers can find other way to get electricity then the product is forget about in monopoly and monopoly cannot exist when there is a competition or any substitute product.
4. 4. 1 Within the perfect competition, the role of non-price competition is insignificant since many sellers sell the merchandise at a fixed price and furthermore, the merchandise are identical. On the other hand, non-price competition may also be referred as offering cost which is the expenses which used to raise the sale of the product of a firm. In perfect competition, firms cannot control the price and their goods are homogenous, so there is absolutely no selling cost. For instance, we do not see any advertising campaign in the television or any other press about a product that has no brand.
4. 4. 2 Under monopolistic competition, some non-price competition are present especially in advertising when organizations compete for his or her products rather than the price tag on the product. Companies should use various methods such as advertisements and advertising to appeal to more customers to buy their products.
4. 4. 3 The companies will try to fully capture the market through better marketing campaign and producing high quality products to the clients instead of minimizing the price. This is because when one company reduces the price of the merchandise other firms will observe the same technique. In fact, the oligopoly businesses compete with the other person through non-price competition rather than price competition. They'll try to make smarter products in conditions of the product quality and number so that can make the firm more competitive when compared with other companies.
4. 4. 4 In a monopoly the companies is only going to advertise because of their own image since there is no competition around. There is only one seller for this product, so organizations do not really need to advertise much to raise the sale of the merchandise. Customers will definitely know that firm supply this type of product and they can only just get such product from this firm.
A monopoly is the marketplace where there is only one seller and many purchasers. Furthermore, the product produced has no close substitutes to make them the price manufacturer or the they have the power to regulate the price in the market. There is also the limitation on the entry of new firms to make it does not have any competition. Hence, they advertise is determined by the products sold. A monopoly will only are present when it fits all the characteristics stated.
What is the Crash controversy?


The genetic code of the deadly H1N1 flu human virus has been cracked–a breakthrough that will now help scientists identify where the virus originated from and reveal how it is presently spreading and how it mutates.
Canadian scientists have become the first to genetically sequence the H1N1 flu virus, vital in developing tests for the infection, and drugs and vaccines to treat it.
Dr David Butler–Jones, Canada's chief public health officer, said on Wednesday that researchers at the National Microbiology Laboratory in Winnipeg, Manitoba, genetically sequenced three samples of the virus, isolated from Mexico and the Canadian provinces of Nova Scotia and Ontario.
Officials in the Centers for Disease Control and Prevention (CDC) in Atlanta, US, have said that having the full genetic sequence will help them watch how the virus evolves. Though it will not help stop the spread or treat the disease immediately, Butler–Jones said it's a step forward in understanding how the virus works.
"This is a world's first. The world's knowledge of the H1N1 flu virus has taken a significant step forward," Canada's health minister Leona Aglukkaq said.
Genetic sequencing of the virus has ruled out a mutation to explain why the Mexican cases have been much more severe than elsewhere.
"We are continuing our analysis, but essentially what it appears to suggest is that there is nothing at the genetic level that differentiates this virus that we've got from Mexico and those from Nova Scotia and Ontario. Differences in human genetics could explain why some people fall severely ill while others experience a mild flu," said Dr Frank Plummer, scientific director–general of the National Microbiology Laboratory, said.
Canada's work on the virus has been submitted to GenBank–an international, searchable database–to allow more researchers to have access to the results and benefit from the information.
According to scientists, decoded of the genetic sequence will now allow the development of faster, more accurate tests, using specific viral DNA fragments to prime PCR reactions. The sequence will also permit the synthesis of specific viral proteins. These could be used to induce antibodies in healthy people, which can then be used to treat influenza patients.
Ultimately, viral proteins will be the basis of a vaccine. Genetic sequencing is the process of determining the order of the molecules that make up the DNA in each gene of an organism. This complete genetic blueprint provides important information for researchers studying the virus.
According to World Health Organisation, more Mexicans have died of the virus, but its manifestations have been much more mild in Canada and other infected countries.
Meanwhile, the breakthrough comes a day after another group of scientists from Lawrence Livermore National Laboratory, US, found that the H1N1 flu genes are dissimilar to past pandemics. Some genetic markers of influenza infection severity have been identified from past outbreaks.


text:
require 'spec_helper'

describe 'eldap::test_install' do
  context 'When OpenLDAP is instanlled and all the fake users are created' do
    let(:chef_run) do
      runner = ChefSpec::ServerRunner.new
      runner.converge(described_recipe)
    end

    it 'converges successfully' do
      expect { chef_run }.to_not raise_error
    end
  end
end

redpajama_set_name: RedPajamaGithub


No matter which type of business you are dealing in, one thing is clear that, all verticals requires efficient management ERP system. Whenever, conversation begins about installing an ERP software, the point of ERP implementation also rise in between which is the most important part of system installation. Implementing an ERP is an extremely critical task for everyone and needs proper skills and experience to complete it on given time frames.
At ACGIL, we a maintained team of implementors who have immense experience in their respective domain. These implementors take care of all software installation related concerns such as demonstration and cost management that have to be handled carefully. In an organization, ERP implementation requires efforts & commitment from their vendors, so that the one who buy an ERP can have effective services for long time.
ERP implementation in India must be done by following a proper ERP implementation process or hierarchy in order to reduce flaws and faults found in the future. The best software installation practices have to be include while selecting and deploying a management system in an organization. The most common and important point of our firm is that we have a dexterous team to complete this task without which nothing would be possible. Now it is the team responsibility to go to the client end and solve system integration problem of the newly installed ERP solution.
ACG Infotech is a reputed information technology based organization that only believes in providing collective results for different relevant ERP facts. Due to our quality assured software products and effective after sales support, we are counted as the best name for ERP system implementation. Moreover, by our services more and more customers are getting involved and find our services beneficial for their constant business growth.


text:
Indonesia: The Reformer of Wringin Sukowono
IWE-WELDD
In Indonesia's East Java Jember District, Najma Milla, a young lady in the village of Wringin Sukowono is steadily making waves.
Ms. Milla is an alumna of RAHIMA (one of WELDD's partners in Indonesia), which is an organization that advocates women's equality based on modern and democratic ideas, and emphasizes its presence on a grassroots level. Ms. Milla is a breath of fresh air: with her husband Nurul, she is establishing an Islamic public-cum-boarding school (pesantren) not solely based on religion. Her pesantren would be the first school of its kind in the village.
Since they came up with the idea in 2006, they have faced a number of barriers: parents, local religious scholars and the public have strongly opposed the idea, dismissing their concept of education dangerous, arguing that it would destroy the religiosity of the younger generation.
"My family thinks of me as bringing a disease, because of my unusual ideas of establishing formal education," says Milla. "When I first established a school here, they said: here comes the troublemaker."
Her school system provides a middle school, and also religious and primary education right in the village. This close proximity to families and their homes mean that parents can no longer use distance as an excuse, and keep their daughters from attending so they can get them married. One of Ms. Milla's main aims is to reduce the rate of child marriage, which she has indeed been successful in. "I always try to make them capable of refusing their parents. At least to be able to give strong reasons to their parents," she says.
According to the 1974 Marriage Law, the minimum legal age of marriage for girls is 16 with parental consent. Despite this, "we see so many students not finishing school because of early marriage," Ms. Milla remarks. "I see lots of students marrying instead of graduating…it makes me sad." Around her, somber parents bow their heads in recognition.
Her school provides a fundamental structural difference from the rest of the schools in the area: boys and girls learn together in her classrooms. This way, they have ideas of equality ingrained into them; both are provided with the same opportunities, which is something completely new for schools in the area.
Through IWE's work with RAHIMA through the WELDD programme, and RAHIMA's transference of this knowledge to others in the area, Najma was exposed to new concepts of pluralism and women's leadership that is feminist, transformative and sustainable. She then went on to integrate these concepts in the management and teaching methods in her school.
The hope is that she continues to embed these new ways of thinking at this grassroots level, and that the girls she teaches will also go on to embody them in their own lives. This is a starting point for young girls and their parents to realize that they have opportunities available to them that lie outside of the goal of marriage.
This short film documents her and Nurul's struggles and experiences in their mission to establish and advance their school.
kekerasan terhadap perempuan dibenarkan secara budaya (CVAW)
Indonesia, RAHIMA, IWE-WELDD, pesantren, Child Marriage, village, school


Debbie joined Barton Willmore in 2015 following 15 years working in the public sector.
Debbie has over 18 years' experience of socio-economic analysis to inform service provision planning and housing development, specialising in demographic modelling.
As an Associate in the national Development Economics team, Debbie is responsible for producing assessments of the current and future population, she is able to prepare housing need assessments at local authority and settlement level, advising on the scale and associated mix of future housing schemes. Debbie also has expertise in establishing the community and employment needs associated with future housing developments to inform discussions with Local Planning/Education Authorities and is a competent user of Geographic Information Systems (GIS).
Debbie has regularly appeared and led on housing need sessions at Local Plan Examinations on behalf of national housebuilders and land owners.


text:
\section{Introduction}
The remarkable success of Deep Reinforcement leaning (RL) on paper is in sharp contrast with its narrow applicability to real-world problems. Among many other reasons, the most important factor preventing the practical deployment of this framework is perhaps its high sample complexity \citep{botvinick2019reinforcement}. This is a very well-known issue and there is a long list of previous works that in one way or another have tried to alleviate it \citep{kakade2003sample, Mnih15Nature,NEURIPS2018_2de5d166}. Nonetheless, the framework is still far from being useful in practice. Here,
rather than proposing yet another method that tries to solve the problem directly, we present a more pragmatic approach to get around it. Our solution is based on the observation that Deep RL's best results have been obtained in domains like video games \citep{bellemare13arcade, vinyals2019grandmaster} or simulated environments \citep{openaigym,Ganesh2019ReinforcementLF, bellemare2020autonomous} where data collection is  extremely fast. Unfortunately, real-world problems are typically more complex and simulators, if available, are usually very slow \citep{dulac2019challenges}.

In this work, we design lightweight versions of large simulators with the goal of speeding up the overall training process. The method we propose applies to domains where agents only interact with a reduced local part of a larger environment, yet they are indirectly being affected by the global dynamics. Traffic control is one example of such environments. Say, for instance, that we wanted to train an agent to control the traffic lights of a particular intersection in a very large city. To do so we could build a small local simulator that captures only the information that is directly relevant to the agent (traffic density in the neighborhood; \citealt{VanDerPol16NIPSWS}). However, after training, we may find out that an agent that does very well in the small simulator, performs poorly in the real intersection. The performance gap would be caused by a data distribution shift \citep{quionero2009dataset, arjovsky2021out}. Even though the simulator might be able to closely mimic the local dynamics (i.e. cars moving within the intersection), it would fail to account for the interactions of the local neighborhood with the rest of the city. Thus, the agent learns a policy based on certain transition dynamics that turn out to be very different in the real world.
Alternatively, we could try to model the dynamics of a sufficiently large portion of the city, but this would surely result in a very slow simulator.

One important property of the traffic domain is that, although the agent's local problem may be \emph{affected} by many external variables (traffic densities in other parts of the city), it is only \emph{directly influenced} by the road segments that connect the intersection with the rest of the city. Hence, we can simply monitor the traffic densities at these road segments since, from the agent's local perspective, they summarize the effect of all the external variables. This insight is not specific to the traffic domain. It is in fact common in networked systems (e.g. warehouse commissioning, \citealt{claes2017decentralised}; electrical power grids, \citealt{wang2021multi}; heating systems, \citealt{gupta2021energy}; telecommunication networks, \citealt{suau2021offline}) that interactions between different components occur through a limited number of variables.

Supported by the formal framework of \emph{influence-based abstraction} (IBA) \citep{oliehoek2021sufficient}, we exploit the above property to build local simulators that mirror the response of the global system through the so called \emph{influence predictor}.

\paragraph{Contributions} The main contribution of this paper is the integration of the IBA framework with the Deep RL workflow.
Our experiments reveal, that the combination of  local simulators and influence predictors can considerably accelerate the reinforcement learning process.
We also demonstrate both theoretically and empirically that the memory needs of the influence predictor are fully determined by the agent's memory capacity. Moreover, we study the impact that distribution shifts caused by changes in the agent's policy may have on the influence predictor, and explore how to prevent the model from picking up on spurious correlations that are not invariant across policies. Finally, we investigate the effect of transfer learning and show how inaccurate simulators might also be able to render effective policies.
\section{Related Work}\label{sec:relatedwork}
The problem of sample complexity has been extensively studied by the RL community. Among many others, the most promising solutions are: replaying previous experiences to make more efficient use of the available data \citep{Mnih15Nature, DBLP:journals/corr/SchaulQAS15,NEURIPS2019_1b742ae2}, or learning surrogate models of the environment dynamics \citep{Sutton90integratedarchitectures,NEURIPS2018_2de5d166, schrittwieser2020mastering, moerland2020model}. Yet, these techniques are only effective when provided enough real samples. If not, replay buffers might not be sufficient to obtain good policies and surrogate models might generalize poorly. An alternative is to train agents with synthetic data coming from a simulator. However, most real-world scenarios are excessively complex and simulators, if available, are computationally expensive \citep{dulac2019challenges}. Here we argue that building a simulator of the entire system is often unnecessary. In fact, as we explain in the following sections, in many situations we can get away by just modelling the dynamics around the agent's local neighborhood.

A few prior works have investigated the computational benefits of factorizing large systems into independent local regions \citep{Nair05AAAI,  Varakantham07AAMAS, Kumar11IJCAI, Witwicki11AAMAS}. Unfortunately,  since local regions are often coupled to one another, such factorizations are not always appropriate. Nonetheless, in many cases, the interactions between regions occur through a limited number of variables. Using this property, the theoretical work by \citet{oliehoek2021sufficient} on influence-based abstraction (IBA) describes how to
build influence-augmented local simulators (IALS) of local-POMDPs, which model only the variables in the environment that are directly relevant to the agent while monitoring the response of the rest of the system with the influence prediction.  The problem is, the exact computation of the conditional influence distribution is intractable and we can only try to estimate it from data. \citet{Congeduti21AAMAS} provide theoretical bounds on the value loss when planning with approximate influence predictors. The work by \citet{NEURIPS2020_2e6d9c60} has empirically demonstrated the advantage of this approach to improve the efficiency of online planning in two discrete toy problems.

Here, we extend the method to high dimensional problems and study how to integrate the IBA framework with Deep RL. This has profound implications that do not arise in the planning context, namely the relation between the agent's memory capacity and the history dependence of the influence predictor (Section \ref{sec:history_dependence}), and the problem of off-policy generalization (Section \ref{sec:policy_dependence}). Moreover, while \citeauthor{NEURIPS2020_2e6d9c60} showed that the IALS outperforms the global simulator only when the time budget is limited, our results reveal that the IALS can train policies in a fraction of the time and that these can match the same performance as policies trained on the GS, without imposing any time constraints, and despite the IALS is only approximate.

\section{Preliminaries}\label{sec:background}
Here we introduce the notation used throughout the paper, provide the mathematical definition of the problem, and describe the Influence-based abstraction (IBA) framework \citep{oliehoek2021sufficient}, which gives theoretical support to the method we introduce in Section \ref{sec:ILS}.
\begin{figure*}[ht]
\hspace{12mm}
\includegraphics[width=0.85\textwidth]{figures/wc-pomdp.pdf}
\vskip -0.2in
\caption{Left: A Dynamic Bayesian Network of a Local-POMDP unrolled 3 timesteps. Right: A diagram of the IALS.}
\label{fig:pomdp+ips}
\end{figure*}

\subsection{Problem definition}
As explained in the introduction, we target domains where, although the agent is affected by the global dynamics, it can only observe its local neighborhood. These can be modelled as partially observable Markov decision processes (POMDP) \citep{Kaelbling96JAIR}.

\begin{definition}[POMDP]
A partially observable Markov decision process (POMDP) is a tuple $\langle S,A,T,R,\Omega,O\rangle$ where $S$ is the state space, $A$ is the set of actions, $T$ is the transition probability function, with $T(s_{t+1}|s_t,a_t)$, $R(s_t,a_t)$ defines the immediate reward, $\Omega$ is the observation space and $O$ is the observation probability function, $O(o_{t+1}|s_{t+1})$.
\end{definition}

As usual, the task consists in finding the policy $\pi$ that maximizes the expected discounted sum of rewards \citep{SuttonBarto98}. Since the agent receives only a local observation $o$ of the true state $s$, a policy that is based only on the most recent information can be sub-optimal. In general, the agent is required to keep track of its past actions and observations to make the right action choices. Policies are therefore mappings from the past action-observation history (AOH), $h_t = \langle o_1, a_1 ..., a_{t-1}, o_t\rangle$, to actions, $\pi(a_t|h_t)$.

We consider, in particular, a special case of a POMDP hereinafter referred to as a \emph{Local-POMDP}.
\begin{definition}[Local-POMDP]
\label{def:local-pomdp}
A Local-POMDP is a POMDP where $O$ and $R$ depend only on a subset of state variables $x \in X \subseteq S$, which define the agent's local region. Such that $O(o_{t+1}|s_{t+1}) = \dot{O}(o_{t+1}|x_{t+1})$ and $R(s_t, a_t) = \dot{R}(x_t, a_t)$, where $\dot{O}$ and $\dot{R}$ are the local observation and reward functions.
\end{definition}

Looking at the above definition, one can argue that simulating the transitions $T(s_{t+1}|s_t, a_t)$ of the full set of state variables
is unnecessary, and while doing so is possible in small problems, it might become computationally intractable in large domains. We can instead define a new transition function $\bar{T}$ that models only the local state variables $x$, $\bar{T}(x_{t+1}|x_t, a_t)$.
The problem is that $x_{t+1}$ may still depend on the rest of state variables $s_t$ and thus $\bar{T}$ is not well defined.
Hence, we are forced to keep
track of the action-local-state history (ALSH) $l_t = \langle x_1, a_1 ..., a_{t-1}, x_t\rangle$
such that we can sample the next local state as
\begin{equation}
\small
\begin{split}
    P(x_{t+1}|h_t, a_t) = \sum_{s_t}T(x_{t+1}|s_t, a_t) P(s_t|l_t).
    \label{eq:local_transitions}
\end{split}
\end{equation}
\subsection{Influence-based Abstraction}
With the above formulation, the problem is only partially solved.
We no longer need to simulate the global state transitions, yet we have to maintain a distribution over the full state given the ALSH, $P(s_t|l_t)$, and then calculate the local transitions with $P(x_{t+1}|s_t, a_t)$. As explained in the introduction, in many local POMDP problems, however, only a fraction of the state variables will \emph{directly influence} the local region.

The diagram in Figure \ref{fig:pomdp+ips} is a Dynamic Bayesian Network (DBN) \citep{pearl88, boutilier1999decision} of a local POMDP prototype. The agent's local region,
corresponds to the variables, denoted by $x \in X$, that lie within the red box, $x = \langle x^1,x^2 \rangle$. The diagram also shows the non-local variables, known as influence sources $u \in U \subseteq S \setminus X$, that influence the local region directly. The three dots on the top indicate that there can be, potentially many, other non-local variables in $S$. These are denoted by $y$ and, as shown in the diagram, can only influence $x$ via $u$. As such, given $u_t$, $x_{t+1}$ is conditionally independent of $y_t$, $P(x_{t+1}|x_t,u_t,y_t) = P(x_{t+1}|x_t, u_t)$.
\begin{definition}[IALM] An influence-augmented local Model (IALM) is a POMDP with local states $x_t \in X$, influence sources $u_t \in U$, local observation function $\dot{O}(o_{t+1}|x_{t+1})$, local reward function $\dot{R}(x_t, a_t)$, local transition function $\dot{T}(x_{t+1}|x_t, u_t, a_t)$ and an influence distribution $I(u_t|l_t)$.
\end{definition}

Using the IALM we can simulate the local transitions as
\begin{equation}
\small
\begin{split}
    P(x_{t+1}| l_t, a_t) =
    \sum_{u_t}  \dot{T}(x_{t+1}|x_t, u_t, a_t)I(u_t|l_t).
    \label{eq:IALM}
\end{split}
\end{equation}
Note that, as opposed to the Local-POMDP, the transition function $\dot{T}$ in the IALM is defined purely in terms of the local state variables and the influence sources. Moreover, since $u_t$ \emph{d-separates} \citep{Bishop06book} $x_t$ from $y_t$, the influence distribution is just the conditional probability over $u_t$ instead of the full set of of state variables $s_t$. All in all, this translates into a much more compact, yet \emph{exact} representation of the problem \citep{oliehoek2021sufficient}, which should be computationally much lighter than the original POMDP.

\section{Influence-Augmented Local Simulators for Deep RL}\label{sec:ILS}
In the following, we describe how we make use of the IALM formulation to design lightweight simulators that can speed up the long training times imposed by
 neural network policies. Figure \ref{fig:pomdp+ips} (right) shows a diagram of the influence-augmented local simulator (IALS), which is composed of a \emph{local simulator} and an \emph{approximate influence predictor}.

\paragraph{Local simulator (LS):}
The LS is an abstracted version of the environment that only models a small portion of it. As opposed to a global simulator (GS), which should closely reproduce the dynamics of every state variable, the LS focuses on characterizing the transitions of those variables $x_t$ that the agent directly interacts with, $\dot{T}(x_{t+1}|x_t, u_t, a_t)$. Recall that, in our setting, both the reward $R$ and  observation $O$ functions are defined in terms of $x_t$ and $a_t$.

\paragraph{Approximate influence predictor (AIP): } The AIP monitors the interactions between the local region and the external variables $y_t$ by estimating $I(u_t|l_t)$. Ideally, we would like the approximation to match the true influence distribution. However,
due to combinatorial explosion, computing the exact probability $I(u_t|l_t)$ is generally intractable \cite{oliehoek2021sufficient}, and so we can only try to approximate it using, for instance, a parametric function. We write  $\hat{I}_\theta$ to denote the AIP,
where $\theta$ are the parameters, which need to be learned from data. Replacing the true influence distribution with an approximation implies that we are no longer guaranteed to find the optimal policy \cite{Congeduti21AAMAS}. Nonetheless, as we show in our experiments, it is often worth trading accuracy for computational efficiency.
We model $\hat{I}_\theta$ using a neural network, which we train on a dataset of $N$ samples of the form $(l_n, u_n)$ collected from the GS (see Algorithm \ref{alg:collect} in Appendix \ref{ap:algorithms}).
Since role of the AIP is to estimate the conditional probability of the influence sources $u_t$ given the past ALSH,
we can formulate the task as a classification problem. The neural network is optimized using the expected cross-entropy loss,
\begin{equation}
\small
     L(\hat{I}_\theta) = - \frac{1}{N}\sum_{n=0}^N u_n \log \hat{I}_\theta(u_n|l_n),
\label{eq:loss}
\end{equation}
which minimizes the KL divergence between the true probabilities $I(u_t|l_t)$ and those predicted by $\hat{I}_\theta(u_t|l_t)$\footnote{Please refer to Appendix \ref{ap:imp_details} for more details on the implementation of the AIP.}.
Once the network has been trained, we can simulate trajectories with the IALS (Algorithm \ref{alg:sample} in Appendix \ref{ap:algorithms}). These trajectories can then be used to train policies with any standard Deep RL method \citep{Mnih15Nature, schulman2017proximal}.

In the next two sections we discuss two important considerations when training AIPs for the RL setting. We first show that, since the agent's memory is inevitably finite, we normally will not need to train complicated AIPs that condition on the full ALSH (Section \ref{sec:history_dependence}). In Section \ref{sec:policy_dependence},
we study the impact that distribution shifts caused by changes in the agent's policy may have on the AIP, and explore how to prevent the AIP from picking up on spurious correlations that are not invariant across policies.

\subsection{Finite memory agents and AIP history dependence }\label{sec:history_dependence}

As mentioned before, we can only rely on approximations of $I(u_t|l_t)$ since computing the exact distribution that conditions on the full ALSH is intractable. Unfortunately though, even with the most sophisticated RNNs \cite{hochreiter1997long, Cho2014Learning}, learning long term temporal dependencies is also very difficult. However, it is worth noting that, if capturing long term dependencies is hard for the AIP, it is too for the agent. In fact, it is very common in Deep RL to find policies that have access only to finite memories \cite{Mnih15Nature, Oh16ICML} or that are trained on short sequences \cite{Schmidhuber91NIPS,hausknecht2015deep}. Thus, if the agents memory is finite, one might well wonder whether the extra level of accuracy that an influence predictor which conditions on the full ALSHs provides is needed.
The result below shows that, the history dependence of the influence predictor is,  under mild conditions, determined by the agent's memory capacity.
\begin{restatable}{theorem}{history}
Let $l_{-k:t}$ be a truncated version of $l_t$ containing only the last $k$ action-local-state pairs and let $a_{0:k}$ be the sequence of past actions from time $0$ up to $k$. Let the agent's policy $\bar{\pi}$ be a function of $h_{-k:t}$ $\bar{\pi}(a_t|h_{-k:t})$, where $h_{-k:t}$ is also a truncated version of $h_t$. If for all $u_t$ we have $P(u_t|l_{-k:t},a_{0:k}) = P(u_t|l_{-k:t})$, then an influence predictor that conditions only on $l_{-k:t}$, $\bar{I}(u_t|l_{-k:t})$, is sufficient to guarantee no loss in value\footnote{Note that this refers only to the value of polices of the form $\bar{\pi}(a_t|h_{-k:t})$, which might perform arbitrarily worse than policies that condition on the full AOH, $\pi(a_t|h_t)$ \cite{Singh94ICML}.}.
\label{thm:history_dependence}
\vskip -0.5 in
\end{restatable}
\renewcommand\qedsymbol{}
\begin{proof}
Found in Appendix \ref{ap:proofs}.
\end{proof}
\vskip -0.1 in
Intuitively, a finite memory agent whose policy conditions on the last $k$ elements in the AOH is only be capable of computing an expectation over the action-values given these $k$ elements. In turn, the distribution of $u_t$ given the full ALSH $l_t$, $I(u_t|l_t)$, is effectively ``washed out'' by the same expectation.
The upshot is that the agent's memory capacity limits the temporal dependencies of the influence predictor\footnote{More details on the practical implications of this result are given in Appendix \ref{ap:imp_details}.}, meaning that $I$ may as well condition directly on $l_{-k:t}$ rather than $l_t$. This insight is empirically evaluated in Section \ref{sec:exp_history_dependence}.

\subsection{Off-policy generalization and d-sets}\label{sec:policy_dependence}
Given that the \emph{true influence distribution} conditions on the full ALSH, it is, in principle, independent of the exploratory policy $\pi_0$ that we use to collect the data. Indeed, if we would represent $I(u_t|l_t)$ with a table, we could compute unbiased estimates of the true probabilities $I(u_t|l_t)$ by using any arbitrary policy that visits every ALSH infinitely often. The problem arises when we use function approximators to estimate the influence sources. In particular, if we call  $P^{\pi_0}(l_t,u_t)$ the stationary joint distribution of influence sources and ALSHs induced by the exploratory policy $\pi_0$ that we use to collect the data $D^{\pi_0} = \left\{(l_1,u_1),... (l_N,u_N)\right\}$ from the GS (as described by Algorithm \ref{alg:collect} in Appendix \ref{ap:algorithms}), we can write
\begin{equation}
\small
    \theta^* = \argmin_\theta L(\hat{I}_\theta, D^{\pi_0}),
    \label{eq:loss_datadist}
\end{equation}
where we see that, in fact, the loss in \eqref{eq:loss} depends on $D^{\pi_0}$. Therefore, because we are fitting $\hat{I}_\theta$ to $D^{\pi_0}$, the model will be biased towards $P^{\pi_0}$. This is not so bad considering that we want the influence predictions to be more reliable for those ALSHs that the agent visits more often. However, it can pose a problem when the policy $\pi$ that we train starts deviating from $\pi_0$. In general, we want the AIP to perform  well on any distribution that the agent may experience during training, $\{P^\pi\}_{\pi \in \Pi}$,
where $\Pi$ denotes the set of possible policies. We then rewrite the optimization problem in \eqref{eq:loss_datadist} as
\begin{equation}
\small
    \theta^* = \argmin_\theta \max_{\pi \in \Pi} L(\hat{I}_\theta, D^{\pi})
\label{eq:00D}
\end{equation}
to indicate that we wish to minimize the worst-case loss.

The key issue here is that, before training, we have only access to the exploratory policy $\pi_0$. Equation \eqref{eq:00D} is an instance of a well-studied problem in the field of out-of-distribution generalization \cite{quionero2009dataset,arjovsky2021out}. In principle, according to \citet[Section~3.6.1]{arjovsky2021out}, finding a solution to \eqref{eq:00D} by minimizing \eqref{eq:loss_datadist} requires, \textbf{(i)} $\supp(P^{\pi}) \subseteq \supp(P^{\pi_0})$, \textbf{(ii)} infinite number of samples, and \textbf{(iii)} infinite capacity models.

The first condition is met so long as $\pi_0(a_t | l_t) > 0$ for all $a_t$ and $l_t$. The second and third conditions, on the other hand, can never be fulfilled in practice. On top of that, high-capacity influence predictors are particularly undesirable for our purpose because they are computationally demanding. The consequence of training low-capacity models on finite datasets is that they may pick-up on spurious correlations \cite{Pearl2000Causality} between ALSHs and influence sources\footnote{\label{fn:example}A visual example of a spurious correlation appearing in the traffic problem is given in Appendix \ref{ap:example}.}. These correlations could be just an artifact of $\pi_0$ and may disappear after the policy is updated. One way to prevent the above, is to find a representation of $l_t$ that elicits an invariant predictor of $u_t$ across all $P^\pi$ \cite{peters2016causal,arjovsky2019invariant, krueger2021ood}.
\begin{definition}[invariant predictor]
A subset of variables $d_t$ from $l_t$ elicit and invariant predictor of $u_t$ across policies  $\pi \in \Pi$ if for all $d_t$ in the intersection of supports, $\supp(P^{\pi_1}(d, u)) \cap \supp(P^{\pi_2}(d, u))$, we have
\begin{equation}
\small
    P^{\pi_1}(u_t| d_t) = P^{\pi_2}(u_t|d_t) \quad \text{for all} \quad \pi_1, \pi_2 \in \Pi.
\end{equation}
\end{definition}
The notion of d-separation \cite{Bishop06book} comes in handy here. Given a DBN, such as the one depicted in Figure \ref{fig:pomdp+ips}, one can determine a subset of variables in the ALSH that is sufficient to predict $u_t$ \cite{oliehoek2021sufficient}.
\begin{definition}[d-separating set] The d-separating set (d-set) is a subset of variables $d_t$ from $l_t$, such that influence sources $u_t$ and the remaining parts of the ALSH $l_t \setminus d_t$ are conditionally independent given $d_t$, $ (u_t \ind l_t \setminus d_t \mid d_t) $.  The d-set is said to be minimal $d^*_t$, when no more variables can be removed from $d^*_t$ while the above still holds.
\label{def:dset}
\end{definition}
\begin{restatable}{theorem}{invariant}
A subset of variables $d_t$ from $l_t$ is only guaranteed to elicit an invariant predictor of $u_t$, across all $\pi \in \Pi$, if (i) $d_t$ constitutes a d-separating set and (ii) all policies are functions of the local AOH, $\pi(h_t)$.\label{thm:invariant}
\end{restatable}
\begin{proof}
Found in Appendix \ref{ap:proofs}.
\end{proof}
\vskip -0.1 in
Note that, according to the result above, the full ALSH $l_t$ does elicit an invariant predictor of $u_t$ since it is, by definition, a d-set. Hence, feeding $l_t$ should be sufficient for the model to find stable and invariant correlations. The problem, however, is that in practice, models tend to converge to low-capacity solutions that require little ``effort" to learn (i.e. least-norm solution) \cite{arjovsky2021out}. As such, the representations formed by an influence predictor that is fed the full ALSH may or may not constitute a d-set. The solution we propose is to feed solely a minimal d-set $d_t^*$ into the AIP $\hat{I}_\theta$. Not only does this reduce the dimensionality of the input space but also prevents the AIP from learning shortcuts \cite{geirhos2020shortcut} resulting from spurious correlations that are not stable under policies other than $\pi_0$\footref{fn:example}. Provided we have the DBN of the problem, there are many algorithms available that can help us find a minimal d-set \cite{Acid1996Algorithm, tian1998finding}. If not, some domain knowledge may be sufficient in most cases, to remove a few variables from $l_t$ and prevent confounding.

Working in the joint space $(d_t, u_t)$ instead of $(l_t, u_t)$ has a another positive effect on off-policy generalization. Since $l_t$ includes the agents actions the distribution $P^{\pi_0}(l,u)$ might be arbitrarily far from a distribution in  $\{P^\pi(l,u)\}_{\pi \in \Pi}$. This is problematic because our low-capacity influence predictor may be unable to generalize well to large distribution shifts.

\begin{restatable}{proposition}{dd}\label{prop:distance}
Let $\pi_0$ be the exploratory policy used to collect the dataset. Then, for all $\pi \in \Pi$
\begin{equation*}
\small
\begin{split}
    KL(P^{\pi_0}(d_t, u_t) || P^{\pi}(d_t, u_t)) \leq  KL(P^{\pi_0}(l_t, u_t) || P^{\pi}(l_t, u_t))
\end{split}
\end{equation*}
\end{restatable}
\begin{proof}
Found in Appendix \ref{ap:proofs}.
\end{proof}

In fact, if actions and d-sets are only weakly-coupled the distributions may be very close. In the extreme case, if d-sets are causally independent with respect to the agent's actions, the influence sources can be considered exogenous \cite{Boutilier96UAI,chitnis2020exogenous} and the two distributions are equivalent.
\begin{restatable}{corollary}{equivalent}
Let $a_{0:t}$ be the past sequence of actions from $0$ to $t$. If $P(d_t| \text{do}(a_{o:t}) ) =  P(d_t)$ for all $d_t$ and $a_{0:t}$. Then, for any $\pi_1 \neq \pi_2 : \pi_1,\pi_2 \in \Pi$ we have $P^{\pi_1}(d_t, u_t) = P^{\pi_2}(d_t, u_t)$ even though $P^{\pi_1}(l_t, u_t) \neq P^{\pi_2}(l_t, u_t)$.
\label{prop:independence}
\end{restatable}
\begin{proof}
Found in Appendix \ref{ap:proofs}.
\end{proof}

Of course, in some domains, certain instantiations of the influence sources may only occur when very specific action sequences are followed. In such cases, a finite dataset collected with a random uniform policy will not be sufficient to fully cover the joint space of influence sources and d-sets, and thus the AIP may be unable to generalize well.  If this happens, assuming a GS is available during training, the obvious solution would be to retrain the AIP every certain number of policy updates. Nonetheless, we argue that in most cases,
such as in the two environments explored in this paper,
the situation will be close to that of Corollary \ref{prop:independence}.

\subsection{Sufficiently similar training conditions}
\label{sec:sufficiently_similar}
Up to this point we have discussed the importance of training accurate AIPs.
Nonetheless, here we present a different perspective: to what extent is it possible to achieve near-optimal performance with inaccurate AIPs? As a matter of fact, it is very common in real-life engineering to model individual components in isolation without considering whether they belong to a larger system. Here we will fall short of giving a complete answer but we will make some observations to suggest that agents might not always require the most accurate AIPs. A view that is also supported by our experiments.

First,
if the IALS can produce at least a few observation sequences that are similar to the ones generated by the true environment, an agent with sufficient capacity will be able to learn from those, and perform well in the real world.
Given that one of the premises of our method is the need for accurate LS, even if the influence predictor was entirely random, the chances of getting just a small percentage of useful experiences may be in fact quite high.
Our second argument, is that different influence distributions might still lead to the same, or very similar, best-response \citep{Becker03AAMAS}. That is, even if the agent is trained on an inaccurate IALS, some of the strategies that the agent will develop might be transferable and could also be useful when followed under real trajectories \citep{lazaric2012transfer}.

\section{Experiments}\label{sec:experiments}

The performance of our method is empirically evaluated on two benchmark domains: traffic control and warehouse commissioning. The goal of the experiments is to:
(1) Study whether we can reduce training times by replacing the GS with the IALS. (2) Measure the impact of the AIP accuracy on the learning process and the agent's final performance.
(3) Investigate the memory needs of the AIP when the agent's memory is finite.

\subsection{Experimental setup}
Agents are trained separately with PPO \cite{schulman2017proximal} on (1) the global simulator (GS), (2) the influence-augmented local simulator (IALS) with an AIP trained offline on a dataset collected from the GS, (3) an IALS that uses an untrained AIP to make predictions (untrained-IALS).
To measure the agent's performance, training is interleaved with periodic evaluations on the GS. The results are averaged over 5 runs with different random seeds.
\begin{figure}
\begin{center}
\centerline{\includegraphics[width=.65\columnwidth]{figures/traffic_grid_intersections12.pdf}}
\caption{A screenshot of the traffic environment.}
\label{fig:trafficgrid}
\end{center}
\vskip-0.3in
\end{figure}
\subsection{Traffic control}\label{sec:traffic}
Figure \ref{fig:trafficgrid} shows a grid-like traffic network composed of 25 intersections. The agent controls the traffic lights at one of the intersections only. The rest of the traffic lights are controlled by fixed actuators that use sensors to adapt to the traffic. The policies used in this experiment for the actuated traffic light controllers were extensively optimized by \citet{wu2017flow}. We train agents separately on the two intersections highlighted in Figure \ref{fig:trafficgrid}. The goal is to maximize the average speed of cars within the intersection. The agent can only observe cars inside the dashed boxes.
\subsubsection{GS, LS, AIP and D-SET}
The GS  and LS are built using Flow \citep{wu2017flow} and SUMO \citep{SUMO2018}. The GS simulates the entire traffic grid (Figure \ref{fig:trafficgrid}) while the LS only models the local neighborhood of the intersection being trained (Figure \ref{fig:local_sim} in Appendix \ref{ap:local_sim}). The influence sources $u_t$  are binary variables indicating whether or not a car will be entering the simulation from each of the four incoming lanes at the current timestep. The AIP $\hat{I}_\theta$ is a feedforward neural network that we train offline on a dataset of $(d_t, u_t)$ pairs collected from the GS. The  d-set $d_t$ is a length 37 binary vector encoding the location of cars along the four incoming lanes. Traffic light information is not included in $d_t$ to prevent confounding\footnote{A visual example of a spurious correlation appearing in the
traffic problem is given in Appendix \ref{ap:example}.}(Section \ref{sec:policy_dependence}). Since the two intersections highlighted in figure \ref{fig:trafficgrid} are influenced differently by the rest of the traffic network we trained separate AIPs for each of them.
\subsubsection{Results}
\begin{figure}
\begin{center}
\centerline{\includegraphics[width=1.0\columnwidth]{figures/traffic_plot.pdf}}
\end{center}
\vskip -0.2in
\caption{\textbf{Top:} Learning curves of agents trained with the GS, the IALS and the untrained-IALS on intersection 1 (Figure \ref{fig:trafficgrid}) as a function of wall-clock time.
\textbf{Middle:} Total runtime of training for 2M training steps on the three simulators. \textbf{Bottom:} Cross entropy loss for the trained and untrained AIPs.}
\vskip -0.1in
\label{fig:traffic_curves}
\end{figure}
The plot at the top of Figure \ref{fig:traffic_curves} are the learning curves of agents trained with the GS, the IALS, and the untrained-IALS to control the traffic lights at intersection 1 (Figure \ref{fig:trafficgrid})\footnote{
Results for intersection 2 are provided in Appendix \ref{ap:intersection2}.}. The plot shows the mean episodic reward as a function of real wall-clock time. Agents are trained for 2M timesteps on all three simulators. The dotted horizontal lines at the end of the red and blue curves show the agent's final performance. The short horizontal line at the beginning of the red curve represents to the AIP's training time.  The black horizontal line indicates the performance of the actuated traffic light controller. The two bar charts at the bottom show the total training time when using each of the three simulators, and the AIP's accuracy with and without training. The results suggest that policies trained on the IALS (red) can match the performance of those trained on the GS (orange) in about $1/3$ of the total training time, despite the IALS is not as accurate as the GS. This is in line with our hypothesis in Section \ref{sec:sufficiently_similar}. Similar influence distributions, $I(u_t|l_t) \approx \hat{I}_\theta(u_t|l_t)$, may lead to the same or very similar local best response. More experiments exploring this phenomenon are provided in Appendix \ref{ap:similar_training_results}. On the other hand, since the distribution $P^\pi(l_t, u_t)$ induced by the untrained AIP is very different from the true distribution, as evidenced by the high cross entropy loss (blue bar bottom chart), agents trained on the untrained-IALS (blue) perform much worse.

\subsection{Warehouse commissioning}
A team of $36$ robots (blue) need to fetch the items (yellow) that appear with probability $0.02$ on the shelves (dashed black lines) of the warehouse in Figure \ref{fig:warehouse}. Each robot has been designated a $5 \times 5$ square region and can only collect the items that appear on the shelves at the edges. The regions overlap so that each of the $4$ item shelves in a robot's region is shared with one of its $4$ neighbors. The blue robots have been programmed to go for the oldest item in their region. The purple robot that is inside the region highlighted by the red box, on the other hand, still needs to be trained. This robot receives as observations a bitmap encoding its own location and a set of $12$ binary variables that indicate whether or not each of the $12$ items within its region needs to be collected. The purple robot, however, cannot see the location of the other robots even though all of them are directly or indirectly influencing it through their actions.
\begin{figure}
\begin{center}
\centerline{\includegraphics[width=.65\columnwidth]{figures/warehouse.png}}
\vskip -0.2in
\caption{A screenshot of the warehouse environment.}
\label{fig:warehouse}
\end{center}
\vskip -0.4in
\end{figure}
\subsubsection{GS, LS, AIP and D-SET}
The GS simulates the entire warehouse (Figure \ref{fig:warehouse}), while the LS models only the $5 \times 5$ square region delimited by the red box (Figure \ref{fig:local_sim} in Appendix \ref{ap:local_sim}). The influence sources $u_t$ encode the location of the four neighbor robots. The AIP is a GRU \citep{Cho2014Learning} that we train offline on a dataset of $(d_t, u_t)$ pairs collected from the GS. If the AIP predicts that any of the neighbor robots is at one of the $12$ cells within the red box and there is an active item on that cell, that item is removed and the purple robot can no longer collect it.  The d-set $d_t$ includes the history of the 12 item variables and 12 additional binary variables encoding whether or not the controlled robot was (is) at one of the item locations. The latter variables are meant to differentiate between an item that is gone because the controlled robot collected it from an item that was picked up by the neighbor robots. The rest of variables in $l_t$ (i.e. the robot's history of locations) are unnecessary for predicting $u_t$, and thus susceptible of becoming confounders (Section \ref{sec:policy_dependence}).

\subsubsection{Results}
The plot at the top of Figure \ref{fig:warehouse_curves} shows the learning curves of the warehouse robot as a function of real wall-clock time when trained on the GS and the two local simulators, IALS and untrained-IALS. Agents are trained for 2M timesteps on all three simulators. The dotted horizontal lines at the end of the red and blue curves show the agent's final performance. The short horizontal line at the beggining of the red curve represents the AIP's training time. The two bar charts at the bottom show the total training time when using each of the three simulators, and the AIP's accuracy with and without training. Again, we see that robots trained on the IALS (red) are able to reach the same performance as those trained on the GS (orange) in about $1/3$ of the total training time despite the IALS is only approximate. Moreover, robots trained on the untrained-IALS (blue) perform reasonable well on the GS. Although the frequency at which items disappear with the untrained-IALS differs very much from that of the true environment, the basic strategy on how to collect items can still be learned. These results further confirm our hypothesis that inaccurate simulators may, in some cases, render effective policies (Section \ref{sec:sufficiently_similar}). More experiments experiments exploring this phenomenon are provided in Appendix \ref{ap:similar_training_results}.
\begin{figure}[tb]
\vskip -0.1in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{figures/warehouse_plot.pdf}}
\caption{\textbf{Top:} Learning curves of agents trained with the GS, the IALS and the untrained-IALS on the the warehouse environment as a function of wall-clock time.
\textbf{Middle:} Total runtime of training for 2M steps on the three simulators. \textbf{Bottom:} Cross entropy loss for the trained and untrained AIPs.}
\label{fig:warehouse_curves}
\end{center}
\vskip -0.4in
\end{figure}
\subsection{Finite memory agents and AIP history dependence}\label{sec:exp_history_dependence}
Here we investigate whether our theoretical result from Theorem \ref{thm:history_dependence} also holds in practice. We want to show that when the agent's memory is finite, meaning it can only access memories from $k$ timesteps in the past, an influence predictor which conditions on the same history length is sufficient. We test this on the warehouse domain. To make the need for memory more evident, we modify the environment so that items always disappear from the robot's region after exactly 8 timesteps.
We first train, AIPs with and without memory. We call the resulting simulators M-IALS and NM-IALS respectively. A histogram showing for how many timesteps items are active before disappearing under each of the simulators is shown in Figure \ref{fig:history_dependence}. As expected, while the former can reach an accuracy of $100\%$ (items always remain for 8 timesteps with the M-IALS), the spectrum is much wider for the NM-IALS. This is because the latter can only estimate the marginal distribution $P^\pi(u_t|o_t)$. Then, we train agents with (M) and without memory (NM) on the M-IALS and the NM-IALS. The results for all four combinations are shown in Figure \ref{fig:history_dependence}. As indicated by the theory, the plot shows that when agents have no memory AIPs may condition only on the current observation (red). In such cases, the extra level of detail that a more accurate history-dependent AIP can provide is wasted (green). In contrast, when agents can distinguish observations from one another based on their memories from the past, AIPs that make predictions by looking at the history are fundamental. This is evidenced by the gap between the blue and orange curves.
\begin{figure}[tb]
\vskip -0.2in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{figures/history_dependence.pdf}}
\vskip -0.2in
\caption{\textbf{Top:} Learning curves of agents with (M) and without memory (NM) trained on M-IALS and NM-IALS. \textbf{Bottom:} item disappearance frequencies with NM-IALS and with M-IALS.}
\label{fig:history_dependence}
\end{center}
\vskip -0.4in
\end{figure}
\section{Conclusion}\label{sec:conclusion}
This paper has offered a practical solution to allow the application of Deep RL methods to large systems, where performing exhaustive simulations can not be afforded. We focused on domains where, although the agent only interacts with a local portion of the environment, it is influenced by the global dynamics. The main idea was to replace the computationally inefficient global simulator by a lightweight version that only models the agent's local problem. However, as we showed in our experiments, directly doing this sometimes translates in a distribution shift on the agent's experience that yields poor performing policies. A good simulator needs to account for the interactions between the local region and the global dynamics. The results of our experiments suggested that by combining a pretrained influence predictor with the local simulator, we could speed up the learning process considerably while matching the performance of agents trained on the global simulator. Moreover, we analyzed the consequences of training influence predictors on data distributions that are different from those the predictor sees when deployed and resolved that, when possible, the human designer should remove from the input those variables that are irrelevant for predicting the influence sources.
Finally, in line with our theoretical results in Section \ref{sec:history_dependence}, our last experiment revealed that the agent's memory capacity limits the memory needs of the influence predictor.

\section*{Acknowledgements}
\newlength{\tmplength}
\setlength{\tmplength}{\columnsep}
This project received funding from the European Research
\begin{wrapfigure}{r}{0.3\columnwidth}
    \vspace{-8pt}
    \hspace{-14pt}
    \includegraphics[width=0.35\columnwidth]{figures/LOGO_ERC-FLAG_EU-cropped}
\end{wrapfigure}Council (ERC)
under the European Union's Horizon 2020  research
and innovation programme (grant agreement No.~758824 \textemdash INFLUENCE).
\setlength{\columnsep}{\tmplength}

redpajama_set_name: RedPajamaArXiv