text:
Michelle McGrath
By Woodley Smith | August 16, 2022
Michelle joined the American Cider Association as executive director in July of 2016. Her rich history includes non-profit management, membership development, supporting farmers and value-added producers, grassroots lobbying, teaching and more. Michelle grew up on a Christmas tree farm in the Sierra Nevada Foothills near the famous Apple Hill. Today she lives in Portland, Oregon with her husband and their two young sons. Michelle has a master's degree in Ecology from University of California, San Diego and a certificate in sustainable business from Pinchot University. She was recognized by Wine Enthusiast Magazine as one of 40 tastemakers under 40 in 2018 and as a SevenFifty Daily Innovator in 2019.
Filter Certified Pommeliers By State

redpajama_set_name: RedPajamaCommonCrawl


text:
Sports advocates sue Dept. of Education over Title IX
In one of the first major challenges of the application of Title IX's three-point test in high schools, the American Sports Council (ASC) filed a lawsuit Thursday against the U.S. Department of Education (EdD).
The federal lawsuit argues that the department's use of gender quotas in high school athletic programs is a violation of the Constitution's Equal Protection Clause.
According to the ASC, formerly the College Sports Council, Title IX activists have been "intimidating" high school districts that they say do not have an adequate "gender balance" in their athletic departments.
Nationally, 1.3 million more boys than girls play high school sports. ASC believes compliance with the current interpretation of the law would result in those boys losing their opportunity to continue playing.
"Not only is this interpretation not supported by law, it has the potential of destroying much of what is so good about the uniquely American athletic system — one that produces the world's best scholar-athletes," said ASC Chairman, Eric Pearson. "This pattern of legal intimidation needs to stop."
Pacific Legal Foundation (PLF) filed the suit on behalf of ASC in the U.S. District Court for the District of Columbia, arguing that the quota system is unconstitutional. (Anti-obesity programs fail, so feds try again)
"Title IX does not require sex-based quotas in high school sports. Quota mandates are at odds with the U.S. Constitution's guarantee of equal protection and equal treatment of people regardless of sex," said PLF attorney Joshua Thompson. "But in recent years, federal policy guidelines on Title IX have made vague references to high schools in a way that allows pro-quota activists to misconstrue Title IX and wrongly demand that high schools institute quota policies."
In 2007, the EdD rejected PLF's petition to "revise or repeal" the current iteration of Title IX to clarify that it does not mandate quotas for high school sports. Thursday's suit alleges that the EdD's rejection of PLF's initial petition was "arbitrary and capricious."
PLF also argues that the department must clarify or repeal the law to clarify that it does not require high school districts to adhere to a quota system for athletics.
The need for clarification comes in the wake of objections from the feminist National Women's Law Center (NWLC), which filed complaints with EdD's Office of Civil Rights, charging that 12 school districts were discriminating against female athletes because of the lack of balance in the number of girls and boys playing sports.
ASC clarified to TheDC that while the lawsuit challenges the manner in which Title IX has been interpreted, it is supportive of the law as it is written.
Tags : department of education


text:
'Gods' is nothing to worship
At 225 minutes, only the most ardent Civil War buffs will make it out alive.
By David Sterritt
"It is well that war is so terrible, or we should grow too fond of it," says Confederate leader Robert E. Lee in "Gods and Generals," the new Civil War movie.
This is a strange remark. On its face, it appears to suggest that sending men into battle is lots of fun, so military officers would do it all the time if not for collateral damages like agony and death. Whatever the general meant, it's an observation worth pondering in light of warfare's vast role in human affairs.
"Gods and Generals" doesn't put much stock in thinking about war, though. It's more interested in highlighting the spectacle of battle and celebrating the men who fought America's bloodiest conflict.
As the title hints, it places "small-g" gods and Civil War generals on about the same moral plane - beings that have their flaws, but stand so far above us ordinary mortals that we can't go wrong by indiscriminately worshiping them.
At times, the distinction between them gets lost altogether, as when we view the death of Thomas "Stonewall" Jackson from a heavenly camera angle, soothed by sentimental lighting and angelic voices. Forget the history books portraying Jackson as an energetic, even enthusiastic killer. Gods and generals are special.
This is the second installment in Ronald F. Maxwell's projected Civil War trilogy, and at 225 minutes long, it feels like a trilogy in itself. That wouldn't be a problem if it had energy and imagination, but those qualities are missing, as is any sense of historical or philosophical context.
Nor does the acting help. Robert Duvall gets star billing, but he's hardly in the film, and when he does show up, he sounds as if he's reading from cue cards. Jeff Daniels is equally bad. The movie's technical quality is inexcusably lax. And how could such a lengthy tale of the American South find time for only two black characters, both grotesquely one-dimensional stereotypes?
Hollywood has produced many dubious depictions of the Civil War, starting with "The Birth of a Nation" in 1915, and this stands with the sorriest of the lot.
• Rated PG-13; contains violence.
For women in law, 'RBG' is their superhero movie
Star Wars Day: What J.J. Abrams can learn from 'The Phantom Menace'
'The Birth of a Nation' is heavy-handed and hero worshiping


Phosphates find uses in meat, fish, dairy and bakery sectors due to their many functions. All phosphates present in food are a nutritive source of phosphorus. A range of sodium and potassium phosphates are available, including blended phosphates for specialised applications in food processing.
Sodium Acid Pyrophosphate prevents the mineral struvite forming in canned seafood. It is also used as a controlled leavening/raising agent in bakery products.
Sodium Tripolyphosphate is used as a texture modifier of vegetables and noodles, an emulsion stabiliser, and a deshelling aid for peanuts and prawns.
Tripotassium Phosphate is used as a sodium-free alternative to trisodium phosphate in low salt/low sodium processed foods.
Tetrasodium pyrophosphate promotes the binding of proteins to water in chicken nuggets, soy-based meat alternatives, imitation crab and lobster products. It is an emulsifier and a thickening agent in instant puddings.
Sodium hexametaphosphate is a multifunctional ingredient used largely in the processed meats, dairy and fish sectors for improved texture and stabilisation.
Dipotassium phosphate is used as a buffering agent, sequestrant, and emulsifier in dairy products. It is used in coffee creamer, cheeses and in meat processing.


text:
Blakeley White-McGuire and Daniel Fetecua in "The Tongue of Flame" Photo by Paula Lobo
HELEN SIMONEAU || THE PLAYGROUND NYC
THE PLAYGROUND NYC
Gibney Dance: Agnes Varis Performing Arts Center
280 Broadway Entrance 53A Chambers
Monday, July 23, 2018 - 10:00am
Wednesday, July 25, 2018 - 10:00am
Friday, July 27, 2018 - 10:00am
www.theplaygroundnyc.org
HELEN SIMONEAU
JULY 23-25-27
CLASS COST:
DROP IN: $20
Gibney Dance: Agnes Varis Performing Arts Center, 280 Broadway NYC
Entrance 53A Chambers
The Playground NYC allows dancers to experience various choreographic styles and ideas, engage with a new community of dancers, and gather resourceful information and inspiration from the array of choreographers involved. In turn, established choreographers are paid for their creative investigations, are given free space to explore new movement with professional dancers, and are provided with the opportunity to establish relationships with new dancers and fellow choreographers.
HELEN SIMONEAU, a native of Rimouski, Québec is described as "a Choreographer-On-The-Rise" with a style that is both "athletic and smooth" — Dance Magazine. Simoneau has been commissioned by The Juilliard School, Oregon Ballet Theatre, the American Dance Festival, UNC School of the Arts, The Yard, Springboard Danse Montréal, and the Swiss International Coaching Project (SiWiC) in Zurich. She was a resident artist at Baryshnikov Arts Center, Bates Dance Festival and has received fellowships from The NYU Center for Ballet and the Arts, the Bogliasco Foundation, and twice from the North Carolina Arts Council. Her company Helen Simoneau Danse has been presented in Austria, Brazil, Canada, France, Greece, Italy, Spain, Switzerland, and has toured throughout Germany, Asia, and the United States. Notable venues that have presented her work include The Guggenheim Museum (NYC), Dance Place (DC), Joyce SoHo (NYC), Tangente (Montréal), The Aoyama Round Theatre (Tokyo), the L.I.G. Art Hall Busan (South Korea), Jacob's Pillow Inside/Out (MA), PACT-Zollverein in Essen (Germany), and Athens International Dance Festival (Greece). Her work was also presented at the 13th Internationales Solo-Tanz-Theatre Festival in Stuttgart, Germany, where she was awarded 1st place for Choreography.
Photo Credit: TODD TURNER


Author Erin Fry Sosne's son in 2015 receiving the rotavirus oral vaccine (l) and pictured today with his mom (r). Photos courtesy of the author.
Editor's note: Why is measles back? In 2000, the U.S. Centers for Disease Control (CDC) declared that America had eradicated measles. Nearly twenty years later, measles outbreaks are popping up around the country, as they have several times since 2000. In mid-March, the CDC confirmed 268 cases of measles in 15 states, and the U.S. Congress is holding hearings to discuss the problem. Scientists, global health policy organizations, and tech companies are speaking in one voice about the dangers of misinformation and resulting vaccine hesitance among some parents.
PATH first ran a version of this reflection piece four years ago on our DefeatDD website, but unfortunately, the article remains just as relevant today as it was then. Author Erin Fry Sosne is the Deputy Director of PATH's advocacy department and has worked with U.S. policymakers, United Nations agencies, technical experts, and civil society groups to promote access to vaccines around the world, including strategies for overcoming the challenge of hesitancy from concerned parents.
Admiration: for the parents who walk miles, wait hours and take time away from work to protect their babies through the power of vaccination; for the science that made it possible to prevent these diseases; and for the complex systems that bring the vaccines safely from their point of manufacture to their place of delivery (a much more complex challenge than one might realize).
The decision not to vaccinate an otherwise healthy child should not be taken lightly; it puts the health of the entire community at risk.
Frustration: that the voice of scientists on the topic of vaccine safety is sometimes overshadowed by passionate individuals with strongly held beliefs based upon anecdotal evidence and conspiracy theories. Vaccines are one of the most studied health interventions of all time. These little miracles in a vial take decades to develop and test for safety and efficacy, and require hundreds of thousands of people around the world working together to ensure the same products reach babies in Albequerque and Ouagadougou. The decision not to vaccinate an otherwise healthy child should not be taken lightly; it puts the health of the entire community at risk.
Pride: to work among scientists and public health professionals, including those at PATH, who help to develop and deliver these lifesaving interventions. To advocate in partnership with political leaders in capitals around the globe who have committed to immunization as one of the best buys in health, dedicating budget for national and local vaccination programs and through financial contributions to organizations such as Gavi, the Vaccine Alliance, UNICEF, and WHO who help make vaccines more affordable and accessible in countries who can't afford them.
I am grateful that my son joined the millions of children who will be vaccinated this year. I am glad that the unfortunate news of the vaccine refusers has stimulated discussion about the value of vaccines among American families, and hope that we can all learn from this tragic outbreak the importance of immunizing ourselves and continuing to make existing and to-be-developed vaccines available to families around the world. While it was hard to watch the jabs go into my son's tiny little thighs, when I swaddled him in my arms, I knew the brief moments of discomfort were worth the shield of protection provided to his little body and the greater community around us.


text:
Most kod Beške preko Dunava je sagrađen u okviru projekta auto-puta Beograd—Novi Sad—Subotica.
Ukupna dužina mosta je 2.205 -{m}-, a širina 14,4 -{m}-. Most ima tri trake za drumski saobraćaj (pun profil za jednu stranu auto-puta) i dve pešačke staze širine 1,7 -{m}-.

Usled velike razlike u nadmorskoj visini leve i desne obale Dunava, most je dizajniran sa nagibom od 2,3% što je ujedno i najveći nagib na auto-putu. Tokom srednjeg vodostaja on se uzdiže 51 -{m}- iznad Dunava.

Glavni dizajner mosta je bio Branko Žeželj, a izgradila ga je Mostogradnja između 1971—1975.

Most se nalazi u blizini mesta Beška po čemu je i dobio naziv.
Investitor je bio SIZ za puteve Vojvodine.
Bombardovan je dva puta i delimično uništen tokom NATO bombardovanja Srbije 1. aprila i 21. aprila 1999, ali je popravljen ubrzo nakon završetka bombardovanja i ponovo otvoren 19. jula 1999. jer je važan deo auto-puta E75.

Dana 3. oktobra 2011. je otvoren i drugi most kod Beške i time je posle skoro 20 godina završen auto-put Beograd—Novi Sad—Subotica

Spoljašnje veze
 YU-build Tehnički detalji mosta (na engleskom)

Mostovi u Srbiji
Грађевине у Инђији
Мостови преко Дунава

redpajama_set_name: RedPajamaWikipedia


You know you need it – great content. And that doesn't just mean website content. It means articles, social media, marketing collateral and more.
It can be overwhelming. There's so much to do, and you don't want to lose revenue or credibility because your content looks unprofessional.
But you can't afford the time it takes to write powerful, personalized content. You need a solution that will satisfy and impress everyone who has a stake in your business.
STOP AGONIZING OVER CONTENT! YOU'VE REACHED THE RIGHT PLACE.
ATW - All Things Writing, LLC is a full-service content development and content marketing company on a mission to help clients shine online and in print.
Coming from private, government and nonprofit sectors, our clients get the kind of content that builds their brand, reputation, image and presence through customized material and personalized services.
You can be one of those clients and benefit from the experience and dedication of ATW and our USA-based partners.
Our proven process of content writing and dissemination not only gives you a strong web presence, it increases brand recognition, as well as boosts your reputation as an expert in your field. All content is customized and belongs to you. It is posted or published in your name.
Because the content is specific to your organization, excerpts can be used again and again in material such as brochures, rack cards, fliers, magazine articles, newsletters, etc.
If you choose one a blogging package, once we reach a certain number of completed articles, you may opt to publish the collection as a book to be used as a giveaway or even to be sold through Amazon.
Utilizing your content is the most cost effective way to market your business.
We know when we post or publish something in your name, it reflects your organization and you personally. That's why all our partners employ content specialists who are professional, published writers and experienced marketers. We want you to look your best.
Projects, blogs, books, letters, press releases – all get at least two pairs of eyes on them to make sure the product is something you will be proud to call your own.
ATW - All Things Writing, LLC was officially launched in January, 2014 by Katherine Mercurio Gotthardt, who has written since she was a child. Thinking she would operate as a traditional freelance writer and editor, Katherine was pleasantly surprised when she became inundated by requests for services. By 2015, she had so much work, she needed a team. She created a network of local, professional partners ready, willing and able to meet the needs of a diverse clientele.
ATW was nominated for the 2016 Prince William Chamber of Commerce's Emerging Business of the Year Award. In 2017, they were nominated for the Chamber's Outstanding Professional Service Award.
In her creative life, Katherine has been nominated twice for Poet Laureate of the Prince William Area. Learn more at www.KatherineGotthardt.com.


Maintenance expenses are the costs incurred to keep an item in good condition or good working order. When purchasing an item that requires upkeep consumers should consider the initial price tag as well as the item's ongoing maintenance expenses. Such expenses are major reasons why home ownership can be more costly than renting. Sometimes items that are merely leased and not owned, such as a leased car, will require the operator to pay maintenance expenses.
Examples of maintenance expenses for automobiles include gas, oil changes, alignment, tire replacement, brake fluid and car washes. Maintenance expenses for a house include lawn care, plumbing repairs, roof repairs, hazard insurance premiums and replacement of worn out appliances.
Responsibility for covering maintenance expenses for rental properties might fall to renters or landlords depending on the upkeep in question. Most rental agreements will define the aspects of a property that renters will be held accountable for. A renter who occupies a beach house for a few days might only be charged for repairs or replacements of any fixtures or appliances that are damaged or destroyed by the renter during their stay. If a renter occupies a home for an extended period, she may be responsible for the regular upkeep of the property. This could include scheduling the repainting of the property or at least alerting the property owner of the need for such work.
Many other maintenance expenses may fall to the landlord to pay for. If the rental residence is furnished, the replacement or repair of articles of furniture is the landlord's responsibility in most cases. The cleaning or replacement of carpeting can be necessary as the residence is frequently used by different renters over time. The clearing out of leaves from rain gutters or the regular trimming of shrubbery would usually be the responsibility of a landlord.
Government requirements can compel a landlord to maintain certain safety and living standards that require frequent stewardship of a property. For example, the heat in an apartment building must meet minimum standards, particularly in the winter.
Infrastructure repairs such as heating and ventilation maintenance can be required in order for a property to be livable. The cost of such regular upkeep is typically paid for by the landlord. Other forms of repair needed on a property can include the repaving of driveways, replacement of windows and insulation, chimney cleaning and tree removal.
A lessee is a person who rents land or property and must follow restrictions and guidelines set by a lease agreement.


text:
Mysterious Myanmar. Part 4: Mandalay
Our last stop in Myanmar was Mandalay. Actually, I've heard this name before, during my school years, even before we learned about the contry called Myanmar in geography lessons, from the song of Robbie Williams. No need to say that I was singing this song in my head the whole time we were on the "Road to Mandalay"…
Though pretty young, founded in the 19th century, Mandalay has more than one million inhabitants what makes it the second biggest city in Myanmar. Mandalay, the last royal capital, is also considered the center of Burmese culture.
The absolute must-do in Mandalay is to climb the famous Mandalay hill before the sunset. The views unveiling during the climb and at the top are incredible. The climb (=walking up the stairs) takes about 45 minutes. Don't forget that you have to climb the hill barefoot: formerly, this was a holy mount. Legend tells that the Buddha, on his visit, had prophesied that a great city would be founded at its foot.
By the way, if you are not a fan of walking or afraid that you don't make it in time before the sunset, you can also take a motorbike taxi to the top what will cost you about 2000 kyats. In Mandalay, this is the quickest and cheapest means of transport.
Probably, you have already seen a picture of hundreds of white pagodas lining up in rows. That is Kuthodaw Pagoda, which lies at the foot of the Mandalay Hill and contains the world's largest book. It was for me a puzzle, what was the connection between the hundreds of pagodas and the book, so I researched a litte bit in the Internet. Here is what I have found: the pagodas are actually housing 730 marble tablets: 729 of them contain body of the text and the last tablet describes the creation of the book. The "pages" are covered on both sides with dense script: each letter had been initially filled with golden ink and the stones themselves were decorated with precious stones such as rubies and diamonds. There 730 marble tables comprise tipitaka, the Theravada Buddhism's religious canon. Unfortunately, after the British invasion in the 1880-s, the troops stolen gold ink and precious stones from the slabs – therefore later the writing was refilled with ordinary black ink keeping the message alive for future generations.
To come back to the city, we took another way – the asphalted road – and were pretty much alone on the road, only from time to time overtaken by the motorbikes. The view to the city was mysterious and bright. The only shortcoming of a lonely walk back were the homeless dogs – though looking harmless, they can become unforeseeable when they get in packs.
Shwenandaw Monastery is another highlight of Mandalay located at the foot of the Mandalay Hill (so you already noticed that all roads lead to this hill!) This monastery was built at the end of the 19th century in the traditional Burmese architectural style. It is famous for its teak carvings of Buddhist myths, which cover its walls and roofs.
In the middle of the city, taking a perfect geometrical shape, spreads the Royal palace, called otherwise "The Famed Royal Emerald Palace" or "Great Golden Royal Palace". It was the last royal palace of the last Burmese monarchy. By the way, this is a replica of the original which was almost completely destroyed during the World War II by bombing.
At daytime, the city center transforms into a buzzing vivid city market: the goods, the sellers, the ambiance is bright and colorful. We were wondering around for couple of hours, just gazing and grasping at the huge variety of exotic stuff.
Another must-see in the Mandalay area is the U-Bein bridge which you can reach with a taxi from Mandalay in about 30 minutes. When you are there, allow for one hour to cross it and get back. Many people say that the most fascinating views reveal during the sunset and sunrise. You can also rent a boat to cross the Taungthaman Lake and take a picture of the oldest and longest teak-wood bridge in the world from the water.
By livetomoveit| 2018-02-10T14:40:52+00:00	December 18th, 2017|
About the Author: livetomoveit
24 hours to explore Kyoto
Tokyo: making the best out of one-week business trip
Two weeks in Sri Lanka: a dream trip. Before you go…
Two weeks in Sri Lanka: a dream trip. Part 6: The Ella – Kandy train
Two weeks in Sri Lanka: a dream trip. Part 5: On the road in Ella
Two weeks in Sri Lanka: a dream trip. Part 4: Safari in Yala
Two weeks in Sri Lanka: a dream trip. Part 3: From Kandy to Sigiriya
Two weeks in Sri Lanka: a dream trip. Part 2: Mirissa, Tangalle and Nilaveli beaches
Two weeks on Sri Lanka: a dream trip. Part 1: from Colombo to Galle
Visiting our sponsoring child on Sri Lanka
Mysterious Myanmar. Part 3: Bagan
Mysterious Myanmar. Part 2: The Dalla Village Scam
Italian Holidays: Bologna, San Marino, Rimini, Modena
Exploring French Antilles
French Antilles: Guadeloupe beaches
French Antilles: Exploring Guadeloupe
French Antilles: places to see in Martinique
livetomoveit on Mysterious Myanmar. Part 2: The Dalla Village Scam
livetomoveit on Caravan travels with baby: our experience
Olga on Caravan travels with baby: our experience
Ramblin' Randy on Mysterious Myanmar. Part 2: The Dalla Village Scam
livetomoveit on Namibia: Etosha National Park
Жизнь в Германии


It is worth describing the details of "remove" since this too is a place where there are engineering and customer decisions to be made. We've already seen one decision which is to make sure we keep the features staged for future use so that a DVD is not required. A second decision is that we also continue to support the APIs available for features where these APIs are necessary to the functionality of Windows or where there are APIs that are used by developers that can be viewed as independent of the component. As many of you know these are often referred to as "dependencies" and with Windows the dependencies can run both internal to Windows and external for ISVs.


text:
Q: Styles not getting rendered in ejs files Styles not getting rendered in ejs files.
Included the stylesheets still they are not getting rendered. Using EJS for rendering.
     <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css" integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
    <link
      rel="stylesheet"
      href="https://bootswatch.com/4/journal/bootstrap.min.css"
    />
    <link
      rel="stylesheet"
      href="css/styles.css"
    />
    <title>Node.js & Passport Login App</title>
    <script
      src="https://code.jquery.com/jquery-3.3.1.slim.min.js"
      integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo"
      crossorigin="anonymous"
    ></script>
    <script
      src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.6/umd/popper.min.js"
      integrity="sha384-wHAiFfRlMFy6i5SRaxvfOCifBUQy1xHdJ/yoi7FRNXMRBu5WHdZYu1hA6ZOblgut"
      crossorigin="anonymous"
    ></script>
    <script
      src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js"
      integrity="sha384-B0UglyR+jN6CkvvICOB2joaf5I4l3gm9GU6Hc1og6Ls7i6U/mkkaduKaBhlAXv9k"
      crossorigin="anonymous"
    ></script>

redpajama_set_name: RedPajamaStackExchange


text:
\section{Introduction} \label{sec:intro}

In the field of reinforcement learning, perhaps the most popular way
to estimate the future discounted reward of states is the method of
\emph{temporal difference learning}.  It is unclear who exactly introduced
this first, however the first explicit version of temporal difference
as a learning rule appears to be Witten \cite{Witten:77}.  The idea is
as follows: The \emph{expected future discounted reward} of a state
$s$ is,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  \overline{V}_s := \mathbf{E} \left \{ r_k + \gamma r_{k+1} + \gamma^2 r_{k+2} + \cdots | s_k = s \right \},
\eeqn
where the rewards $r_k, r_{k+1}, \ldots$ are geometrically discounted
into the future by $\gamma<1$.  From this definition it follows that,
\begin{equation}}    \def\eeq{\end{equation} \label{eqn:trueV2}
  \overline{V}_s = \mathbf{E} \left \{ r_k + \gamma \overline{V}_{s_{k+1}} | s_k = s \right \}.
\eeq
Our task, at time $t$, is to compute an estimate $V^t_s$ of
$\overline{V}_s$ for each state~$s$.  The only information we have to base
this estimate on is the current history of state transitions, $s_1,
s_2, \ldots, s_t$, and the current history of observed rewards, $r_1,
r_2, \ldots, r_t$.  Equation~\req{eqn:trueV2} suggests that at time
$t+1$ the value of $r_t + \gamma V_{s_{t+1}}$ provides us with information
on what $V^t_s$ should be: If it is higher than $V^t_{s_t}$ then
perhaps this estimate should be increased, and vice versa.  This
intuition gives us the following estimation heuristic
for state $s_t$,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  V^{t+1}_{s_t} := V^t_{s_t} + \alpha \left( r_t + \gamma V^t_{s_{t+1}} - V^t_{s_t} \right),
\eeqn
where $\alpha$ is a parameter that controls the rate of learning.
This type of temporal difference learning is known as $\mbox{TD}(0)$.

One shortcoming of this method is that at each time step the value of
only the last state $s_t$ is updated.  States before the last state
are also affected by changes in the last state's value and thus these
could be updated too.  This is what happens with so
called \emph{temporal difference learning with eligibility traces},
where a history, or trace, is kept of which states have been recently
visited.  Under this method, when we update the value of a state we
also go back through the trace updating the earlier states as well.
Formally, for any state $s$ its eligibility trace is computed by,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  E^t_s := \left\{ \begin{array}{ll}
  \gamma \lambda E^{t-1}_s & \textrm{ if $s \neq s_t$,}\\
  \gamma \lambda E^{t-1}_s + 1 & \textrm{ if $s = s_t$,}
\end{array} \right.
\eeqn
where $\lambda$ is used to control the rate at which the eligibility trace
is discounted.  The temporal difference update is then, for all states
$s$,
\begin{equation}}    \def\eeq{\end{equation}\label{eq:tdl}
  V^{t+1}_s := V^t_s + \alpha E^t_s \left( r + \gamma V^t_{s_{t+1}} - V^t_{s_t} \right).
\eeq
This more powerful version of temporal different learning is known as
$\mbox{TD}(\lambda)$ \cite{Sutton:88}.

The main idea of this paper is to derive a temporal difference rule
from statistical principles and compare it to the standard heuristic
described above.
Superficially, our work has some similarities to LSTD($\lambda$)
(\cite{Lagoudakis:03} and references therein).  However LSTD is
concerned with finding a least-squares linear function approximation,
it has not yet been developed for general $\gamma$ {\em and}
$\lambda$, and has update time quadratic in the number of
features/states. On the other hand, our algorithm ``exactly''
coincides with TD/Q/Sarsa($\lambda$) for finite state spaces, but with
a novel learning rate derived from statistical principles. We
therefore focus our comparison on TD/Q/Sarsa.  For a recent survey of
methods to set the learning rate see \cite{george:06}.

In \emph{Section~\ref{sec:derive}} we derive a least squares estimate
for the value function.  By expressing the estimate as an incremental
update rule we obtain a new form of $\mbox{TD}(\lambda)$, which we call
$\mbox{HL}(\lambda)$.
In \emph{Section~\ref{sec:lmp}} we compare $\mbox{HL}(\lambda)$ to $\mbox{TD}(\lambda)$ on a
simple Markov chain.  We then test it on a random Markov chain
in \emph{Section~\ref{sec:rmp}} and a non-stationary environment
in \emph{Section~\ref{sec:nsmp}}.  In
\emph{Section~\ref{sec:mdp}} we derive two new methods for policy learning
based on $\mbox{HL}(\lambda)$, and compare them to Sarsa($\lambda$) and Watkins'
$\mbox{Q}(\lambda)$ on a simple reinforcement learning
problem.  \emph{Section~\ref{sec:con}} ends the paper with a summary
and some thoughts on future research directions.

\section{Derivation}\label{sec:derive}

The \emph{empirical future discounted reward} of a state $s_k$ is the
sum of \emph{actual rewards} following from state $s_k$ in time steps
$k, k+1, \ldots$, where the rewards are discounted as they go into the
future.  Formally, the empirical value of state $s_k$ at time $k$ for
$k=1,...,t$ is,
\begin{equation}}    \def\eeq{\end{equation} \label{eqn:empDef}
v_k := \sum_{u=k}^\infty \gamma^{u-k} r_u,
\eeq
where the future rewards $r_u$ are geometrically discounted by $\gamma<1$.
In practice the exact value of $v_k$ is always unknown to us as it
depends not only on rewards that have been already observed, but also
on unknown future rewards.  Note that if $s_m = s_n$ for $m \neq n$,
that is, we have visited the same state twice at different times $m$
and $n$, this does not imply that $v_n = v_m$ as the observed rewards
following the state visit may be different each time.

Our goal is that for each state $s$ the estimate $V_s^t$ should be as
close as possible to the true expected future discounted reward
$\overline{V}_s$.  Thus, for each state $s$ we would like $V_s$ to be close
to $v_k$ for all $k$ such that $s = s_k$.  Furthermore, in
non-stationary environments we would like to discount old evidence by
some parameter $\lambda \in (0,1]$.  Formally, we want to minimise the
loss function,
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:loss}
L := \frac{1}{2} \sum_{k=1}^t \lambda^{t-k} \big( v_k -V_{s_k}^t \big)^2.
\eeq
For stationary environments we may simply set $\lambda = 1$ a priori.

As we wish to minimise this loss, we take the partial derivative with
respect to the value estimate of each state and set to zero,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  \frac{\partial L}{\partial V_s^t}
  =  - \sum_{k=1}^t \lambda^{t-k} \big( v_k - V_{s_k}^t \big) \delta_{s_ks}\\
  =  V_s^t \sum_{k=1}^t \lambda^{t-k} \delta_{s_ks} - \sum_{k=1}^t \lambda^{t-k} \delta_{s_ks} v_k  = 0,
\eeqn
where we could change $V_{s_k}^t$ into $V_s^t$ due to the presence
of the Kronecker $\delta_{s_ks}$, defined $\delta_{xy} := 1$ if
$x=y$, and 0 otherwise.  By defining a discounted state visit counter
$N_s^t := \sum_{k=1}^t \lambda^{t-k} \delta_{s_k s}$ we get
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:Vt}
  V_s^t N_s^t = \sum_{k=1}^t \lambda^{t-k} \delta_{s_ks} v_k.
\eeq
Since $v_k$ depends on future rewards $r_k$, Equation~\req{eqn:Vt} can
not be used in its current form.
Next we note that $v_k$ has a self-consistency property with respect
to the rewards.  Specifically, the tail of the future discounted
reward sum for each state depends on the empirical value at time $t$
in the following way,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  v_k = \sum_{u=k}^{t-1} \gamma^{u-k} r_u + \gamma^{t-k} v_t.
\eeqn
Substituting this into Equation~\req{eqn:Vt} and exchanging the order
of the double sum,
\begin{eqnarray*}}  \def\eqan{\end{eqnarray*}
  V_s^t N_s^t & = &
  \sum_{u=1}^{t-1} \sum_{k=1}^u \lambda^{t-k} \delta_{s_ks} \gamma^{u-k} r_u
  + \sum_{k=1}^t \lambda^{t-k} \delta_{s_ks} \gamma^{t-k} v_t \\
  & = & \sum_{u=1}^{t-1}  \lambda^{t-u} \sum_{k=1}^u (\lambda \gamma)^{u-k} \delta_{s_ks} r_u
  + \sum_{k=1}^t (\lambda \gamma)^{t-k} \delta_{s_ks} v_t \\
  & = & R_s^t + E_s^t v_t,
\eqan
where $E_s^t := \sum_{k=1}^t (\lambda \gamma)^{t-k} \delta_{s_k s}$ is
the eligibility trace of state $s$, and $R_s^t := \sum_{u=1}^{t-1} \lambda^{t-u}
E_s^u r_u$ is the discounted reward with eligibility.

$E_s^t$ and $R_s^t$ depend only on quantities known at time $t$.
The only unknown quantity is $v_t$,
which we have to replace with our current estimate of this value at
time $t$, which is $V_{s_t}^t$.  In other words, we bootstrap our estimates.
This gives us,
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:Vt2}
  V_s^t N_s^t = R_s^t + E_s^t V_{s_t}^t.
\eeq
For state $s = {s_t}$, this simplifies to
$V_{s_t}^t = R_{s_t}^t /(N_{s_t}^t - E_{s_t}^t)$.
Substituting this back into Equation~\req{eqn:Vt2} we obtain,
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:Vt3}
  V_s^t N_s^t =  R_s^t + E_s^t \frac{R_{s_t}^t}{N_{s_t}^t - E_{s_t}^t}.
\eeq
This gives us an explicit expression for our $V$ estimates.  However,
from an algorithmic perspective an incremental update rule is more
convenient.  To derive this we make use of the relations,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  N_s^{t+1} = \lambda N_s^t + \delta_{{s_{t+1}} s},\qquad
  E_s^{t+1} = \lambda \gamma E_s^t + \delta_{{s_{t+1}} s},\qquad
  R_s^{t+1} = \lambda R_s^t + \lambda E_s^t r_t,
\eeqn
with $N_s^0=E_s^0=R_s^0=0$.
Inserting these into Equation~\req{eqn:Vt3} with $t$ replaced by $t+1$,
\begin{eqnarray*}}  \def\eqan{\end{eqnarray*}
  V_s^{t+1} N_s^{t+1}
  & = & R_s^{t+1} + E_s^{t+1} \frac{R_{s_{t+1}}^{t+1}} {N_{s_{t+1}}^{t+1} - E_{s_{t+1}}^{t+1}}\\
  & = & \lambda R_s^t + \lambda E_s^t r_t +
  E_s^{t+1} \frac{R_{s_{t+1}}^t \! + E_{s_{t+1}}^t r_t}{ N_{s_{t+1}}^t \! - \gamma E_{s_{t+1}}^t}.
\eqan

By solving Equation~\req{eqn:Vt2} for $R_s^t$ and substituting
back in,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  V_s^{t+1} N_s^{t+1} = \lambda \big( V_s^t N_s^t - E_s^t V_{s_t}^t \big) + \lambda E_s^t r_t + E_s^{t+1}
  \frac{N_{s_{t+1}}^t V_{s_{t+1}}^t \! - E_{s_{t+1}}^t V_{s_t}^t \! + E_{s_{t+1}}^t r_t}{N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t}
\eeqn
\begin{displaymath}}\def\eeqn{\end{displaymath}
  = \: \big( \lambda N_s^t + \delta_{\sus} \big) V_s^t
  \! - \delta_{\sus} V_s^t \! - \lambda E_s^t V_{s_t}^t +  \lambda E_s^t r_t
  \phantom{xxxxxx}
\eeqn
\begin{displaymath}}\def\eeqn{\end{displaymath}
  \phantom{xxxxxxxxx} + E_s^{t+1}
  \frac{N_{s_{t+1}}^t V_{s_{t+1}}^t \! -E_{s_{t+1}}^t V_{s_t}^t \! + E_{s_{t+1}}^t r_t}{N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t}.
\eeqn
Dividing through by $N_s^{t+1} (=\lambda N_s^t \! +
\delta_{\sus})$,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  V_s^{t+1} =
  V_s^t
  + \frac{-\delta_{\sus} V_s^t \! - \lambda E_s^t V_{s_t}^t +  \lambda E_s^t r_t}
  {\lambda N_s^t \! + \delta_{\sus}} \phantom{xxxxxxxxxxxxxxx}
\eeqn
\begin{displaymath}}\def\eeqn{\end{displaymath}
  \phantom{xxxxxxxxx} + \frac{(\lambda \gamma E_s^t + \delta_{\sus})
  (N_{s_{t+1}}^t V_{s_{t+1}}^t \! -E_{s_{t+1}}^t V_{s_t}^t \! + E_{s_{t+1}}^t r_t)}
  {(N_{s_{t+1}}^t  - \gamma E_{s_{t+1}}^t) (\lambda N_s^t \! + \delta_{\sus}) }.
\eeqn

Making the first denominator the same as the second, then expanding
the numerator,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  V_s^{t+1} =
  V_s^t + \frac{
  \lambda E^t_s r_t N^t_{s_{t+1}} - \lambda E^t_s V^t_{s_t} N^t_{s_{t+1}} - \delta_{{s_{t+1}} s} V^t_s N^t_{s_{t+1}}
  - \lambda \gamma E^t_{s_{t+1}} E^t_{s} r_t}
  {(N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t) (\lambda N_s^t \! + \delta_{\sus}) } \phantom{xxxxxxxx}
\eeqn
\begin{displaymath}}\def\eeqn{\end{displaymath}
  + \frac{
    \lambda \gamma E^t_{s_{t+1}} E^t_s V^t_{s_t} + \gamma E^t_{s_{t+1}} V^t_s \delta_{{s_{t+1}} s}
  + \lambda \gamma E^t_s N^t_{s_{t+1}} V^t_{s_{t+1}} - \lambda \gamma E^t_s E^t_{s_{t+1}} V^t_{s_t}}
  {(N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t) (\lambda N_s^t \! + \delta_{\sus}) }
\eeqn
\begin{displaymath}}\def\eeqn{\end{displaymath}
  + \frac{\lambda \gamma E^t_s E^t_{s_{t+1}} r_t + \delta_{{s_{t+1}} s} N^t_{s_{t+1}} V^t_{s_{t+1}}
  - \delta_{{s_{t+1}} s} E^t_{s_{t+1}} V^t_{s_t} + \delta_{{s_{t+1}} s} E^t_{s_{t+1}} r_t}
  {(N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t) (\lambda N_s^t \! + \delta_{\sus}) }.
\eeqn
After cancelling equal terms (keeping in mind that in every term with
a Kronecker $\delta_{xy}$ factor we may assume that $x=y$ as the term
is always zero otherwise), and factoring out $E^t_s$ we obtain,
\begin{displaymath}}\def\eeqn{\end{displaymath}
  V_s^{t+1} = V_s^t +
  \frac{ E^t_s \big(
  \lambda r_t N^t_{s_{t+1}} - \lambda V^t_{s_t} N^t_{s_{t+1}} + \gamma V^t_s \delta_{{s_{t+1}} s}
  + \lambda \gamma N^t_{s_{t+1}} V^t_{s_{t+1}} - \delta_{{s_{t+1}} s} V^t_{s_t} + \delta_{{s_{t+1}} s} r_t \big)}
  {(N_{s_{t+1}}^t - \gamma E_{s_{t+1}}^t) (\lambda N_s^t \! + \delta_{\sus}) }
\eeqn
Finally, by factoring out $\lambda N_{s_{t+1}}^t + \delta_{\sus}$ we obtain our
update rule,
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:new}
  V_s^{t+1} = V_s^t \, + \, E_s^t \: \beta_t(s,{s_{t+1}}) \, \big( r_t + \gamma V_{s_{t+1}}^t - V_{s_t}^t \big),
\eeq
where the learning rate is given by,
\begin{equation}}    \def\eeq{\end{equation}\label{eqn:newrate}
  \beta_t(s,{s_{t+1}}) \: := \:
  \frac{1}{N_{s_{t+1}}^t \! - \gamma E_{s_{t+1}}^t}
  \: \: \frac{N_{s_{t+1}}^t}{N_s^t}.
\eeq
Examining Equation~\req{eqn:new}, we find the usual update equation
for temporal difference learning with eligibility traces (see
Equation~\req{eq:tdl}), however the learning rate $\alpha$ has now
been replaced by $\beta_t(s,{s_{t+1}})$.  This learning rate was derived
from statistical principles by minimising the squared loss between the
estimated and true state value.  In the derivation we have exploited
the fact that the latter must be self-consistent and then bootstrapped
to get Equation~\req{eqn:Vt2}.  This gives us an equation for the
learning rate for each state transition at time $t$, as opposed to the
standard temporal difference learning where the learning rate $\alpha$
is either a fixed free parameter for all transitions, or is decreased
over time by some monotonically decreasing function.  In either case,
the learning rate is not automatic and must be experimentally tuned
for good performance.  The above derivation appears to theoretically
solve this problem.

The first term in $\beta_t$ seems to provide some type of
normalisation to the learning rate, though the intuition behind this
is not clear to us.  The meaning of second term however can be
understood as follows: $N^t_s$ measures how often we have visited
state $s$ in the recent past.  Therefore, if $N^t_s \ll N^t_{s_{t+1}}$ then
state $s$ has a value estimate based on relatively few samples, while
state ${s_{t+1}}$ has a value estimate based on relatively many samples.  In
such a situation, the second term in $\beta_t$ boosts the learning
rate so that $V_s^{t+1}$ moves more aggressively towards the
presumably more accurate $r_t + \gamma V_{s_{t+1}}^t$.  In the opposite
situation when ${s_{t+1}}$ is a less visited state, we see that the reverse
occurs and the learning rate is reduced in order to maintain the
existing value of $V_s$.

\section{A simple Markov process} \label{sec:lmp}

For our first test we consider a simple Markov process with 51 states.
In each step the state number is either incremented or decremented by
one with equal probability, unless the system is in state 0 or 50 in
which case it always transitions to state 25 in the following step.
When the state transitions from 0 to 25 a reward of 1.0 is generated,
and for a transition from 50 to 25 a reward of -1.0 is generated.  All
other transitions have a reward of 0.  We set the discount value $\gamma =
0.99$ and then computed the true discounted value of each state by
running a brute force Monte Carlo simulation.

We ran our algorithm 10 times on the above Markov chain and computed
the root mean squared error in the value estimate across the states at
each time step averaged across each run.  The optimal value of $\lambda$
for $\mbox{HL}(\lambda)$ was 1.0, which was to be expected given that the
environment is stationary and thus discounting old experience is not
helpful.

For $\mbox{TD}(\lambda)$ we tried various different learning rates and values of
$\lambda$.  We could find no settings where $\mbox{TD}(\lambda)$ was competitive with
$\mbox{HL}(\lambda)$.  If the learning rate $\alpha$ was set too high the system
would learn as fast as $\mbox{HL}(\lambda)$ briefly before becoming stuck.  With
a lower learning rate the final performance was improved, however the
initial performance was now much worse than $\mbox{HL}(\lambda)$.  The results of
these tests appear in Figure~\ref{graph-1b}.

\begin{figure}
\begin{minipage}[t]{0.49\textwidth}
\includegraphics[width=1.07\textwidth]{graph-1b-big.eps}
\caption{\label{graph-1b}51 state Markov process averaged over 10 runs.
The parameter $a$ is the learning rate $\alpha$.}
\end{minipage}
\hspace{0.02\textwidth}
\begin{minipage}[t]{0.49\textwidth}
\includegraphics[width=1.07\textwidth]{graph-1c-big.eps}
\caption{\label{graph-1c}51 state Markov process averaged over 300 runs.}
\end{minipage}
\end{figure}

Similar tests were performed with larger and smaller Markov chains,
and with different values of~$\gamma$.  $\mbox{HL}(\lambda)$ was consistently
superior to $\mbox{TD}(\lambda)$ across these tests.  One wonders whether this
may be due to the fact that the implicit learning rate that $\mbox{HL}(\lambda)$
uses is not fixed.  To test this we explored the performance of a
number of different learning rate functions on the 51 state Markov
chain described above.  We found that functions of the form
$\frac{\kappa}{t}$ always performed poorly, however good performance
was possible by setting $\kappa$ correctly for functions of the form
$\frac{\kappa}{\sqrt{t}}$ and $\frac{\kappa}{\sqrt[3]{t}}$.  As the
results were much closer, we averaged over 300 runs.  These results
appear in Figure~\ref{graph-1c}.

With a variable learning rate $\mbox{TD}(\lambda)$ is performing much better,
however we were still unable to find an equation that reduced the
learning rate in such a way that $\mbox{TD}(\lambda)$ would outperform $\mbox{HL}(\lambda)$.
This is evidence that $\mbox{HL}(\lambda)$ is adapting the learning rate
optimally without the need for manual equation tuning.

\section{Random Markov process} \label{sec:rmp}

To test on a Markov process with a more complex transition structure,
we created a random 50 state Markov process.  We did this by creating
a 50 by 50 transition matrix where each element was set to 0 with
probability 0.9, and a uniformly random number in the interval $[0,1]$
otherwise.  We then scaled each row to sum to 1.  Then to transition
between states we interpreted the $i^{th}$ row as a probability
distribution over which state follows state $i$.  To compute the
reward associated with each transition we created a random matrix as
above, but without normalising.  We set $\gamma = 0.9$ and then ran a
brute force Monte Carlo simulation to compute the true discounted
value of each state.

The $\lambda$ parameter for $\mbox{HL}(\lambda)$ was simply set to 1.0 as the
environment is stationary.  For $\mbox{TD}$ we experimented with a range of
parameter settings and learning rate decrease functions.  We found
that a fixed learning rate of $\alpha = 0.2$, and a decreasing rate of
$\frac{1.5}{\sqrt[3]{t}}$ performed reasonable well, but never as well
as $\mbox{HL}(\lambda)$.  The results were generated by averaging over 10 runs,
and are shown in Figure~\ref{graph-1d}.

Although the structure of this Markov process is quite different to
that used in the previous experiment, the results are again similar:
$\mbox{HL}(\lambda)$ preforms as well or better than $\mbox{TD}(\lambda)$ from the beginning
to the end of the run.  Furthermore, stability in the error towards
the end of the run is better with $\mbox{HL}(\lambda)$ and no manual learning
tuning was required for these performance gains.

\begin{figure}
\begin{minipage}[t]{0.49\textwidth}
\includegraphics[width=1.07\textwidth]{graph-1d-big.eps}
\caption{\label{graph-1d}Random 50 state Markov process.
The parameter $a$ is the learning rate $\alpha$.}
\end{minipage}
\hspace{0.02\textwidth}
\begin{minipage}[t]{0.49\textwidth}
\includegraphics[width=1.07\textwidth]{graph-2-big.eps}
\caption{\label{graph-2}21 state non-stationary Markov process.}
\end{minipage}
\end{figure}

\section{Non-stationary Markov process} \label{sec:nsmp}

The $\lambda$ parameter in $\mbox{HL}(\lambda)$, introduced in
Equation~\req{eqn:loss}, reduces the importance of old observations
when computing the state value estimates.  When the environment is
stationary this is not useful and so we can set $\lambda = 1.0$, however in
a non-stationary environment we need to reduce this value so that the
state values adapt properly to changes in the environment.  The more
rapidly the environment is changing, the lower we need to make $\lambda$ in
order to more rapidly forget old observations.

To test $\mbox{HL}(\lambda)$ in such a setting, we used the Markov chain from
Section~\ref{sec:lmp}, but reduced its size to 21 states to speed up
convergence.  We used this Markov chain for the first 5,000 time
steps.  At that point, we changed the reward when transitioning from
the last state to middle state to from -1.0 to be 0.5.  At time 10,000
we then switched back to the original Markov chain, and so on
alternating between the models of the environment every 5,000 steps.
At each switch, we also changed the target state values that the
algorithm was trying to estimate to match the current configuration of
the environment.  For this experiment we set $\gamma = 0.9$.

As expected, the optimal value of $\lambda$ for $\mbox{HL}(\lambda)$ fell from 1 down
to about 0.9995.  This is about what we would expect given that each
phase is 5,000 steps long.  For $\mbox{TD}(\lambda)$ the optimal value of $\lambda$
was around 0.8 and the optimum learning rate was around $0.05$.  As we
would expect, for both algorithms when we pushed $\lambda$ above its
optimal value this caused poor performance in the periods following
each switch in the environment (these bad parameter settings are not
shown in the results).  On the other hand, setting $\lambda$ too low
produced initially fast adaption to each environment switch, but poor
performance after that until the next environment change.  To get
accurate statistics we averaged over 200 runs.  The results of these
tests appear in Figure~\ref{graph-2}.

For some reason $\mbox{HL}(0.9995)$ learns faster than $\mbox{TD}(0.8)$ in the
first half of the first cycle, but only equally fast at the start of
each following cycle.  We are not sure why this is happening.  We
could improve the initial speed at which $\mbox{HL}(\lambda)$ learnt in the last
three cycles by reducing $\lambda$, however that comes at a performance
cost in terms of the lowest mean squared error attained at the end of
each cycle.  In any case, in this non-stationary situation $\mbox{HL}(\lambda)$
again performed well.

\section{Windy Gridworld} \label{sec:mdp}

Reinforcement learning algorithms such as Watkins' $\mbox{Q}(\lambda)$
\cite{Watkins:89} and Sarsa($\lambda$) \cite{Rummery:94,Rummery:95} are
based on temporal difference updates.  This suggests that new
reinforcement learning algorithms based on $\mbox{HL}(\lambda)$ should be
possible.

For our first experiment we took the standard Sarsa($\lambda$)
algorithm and modified it in the obvious way to use an HL temporal
difference update.  In the presentation of this algorithm we have
changed notation slightly to make things more consistent with that
typical in reinforcement learning.  Specifically, we have dropped the
$t$ super script as this is implicit in the algorithm specification,
and have defined $Q(s,a) := V_{(s,a)}$, $E(s,a) := E_{(s,a)}$ and
$N(s,a) := N_{(s,a)}$.  Our new reinforcement learning algorithm,
which we call HLS($\lambda$) is given in Algorithm~\ref{alg:hls}.
Essentially the only changes to the standard Sarsa($\lambda$) algorithm
have been to add code to compute the visit counter $N(s,a)$, add a
loop to compute the $\beta$ values, and replace $\alpha$ with $\beta$
in the temporal difference update.

To test HLS($\lambda$) against standard Sarsa($\lambda$) we used the Windy
Gridworld environment described on page 146 of \cite{Sutton:98}.  This
world is a grid of 7 by 10 squares that the agent can move through by
going either up, down, left of right.  If the agent attempts to move
off the grid it simply stays where it is.  The agent starts in the
$4^{th}$ row of the $1^{st}$ column and receives a reward of 1 when it
finds its way to the $4^{th}$ row of the $8^{th}$ column.  To make
things more difficult, there is a ``wind'' blowing the agent up 1 row
in columns 4, 5, 6, and 9, and a strong wind of 2 in columns 7 and 8.
This is illustrated in Figure~\ref{windworld}.  Unlike in the original
version, we have set up this problem to be a continuing discounted
task with an automatic transition from the goal state back to the
start state.

\begin{algorithm}[t]
\caption{HLS($\lambda$)}
\label{alg:hls}
\begin{algorithmic}
\STATE Initialise $Q(s,a)=0$, $N(s,a)=1$ and $E(s,a)=0$ for all $s$, $a$
\STATE Initialise $s$ and $a$
\REPEAT
  \STATE Take action $a$, observed $r$, $s'$
  \STATE Choose $a'$ by using $\epsilon$-greedy selection on $Q(s',\cdot)$
  \STATE $\Delta \leftarrow r + \gamma Q(s',a') - Q(s,a)$
  \STATE $E(s,a) \leftarrow E(s,a)+1$
  \STATE $N(s,a) \leftarrow N(s,a)+1$

  \FORALL {$s,a$}
  \STATE $\beta( (s,a), (s',a') ) \leftarrow \frac{1}{N(s',a') - \gamma E(s',a')} \: \: \frac{N(s',a')}{N(s,a)}$
  \ENDFOR

  \FORALL {$s,a$}
    \STATE $Q(s,a) \leftarrow Q(s,a) + \beta \big( (s,a), (s',a') \big) E(s,a) \Delta$
    \STATE $E(s,a) \leftarrow \gamma \lambda E(s,a)$
    \STATE $N(s,a) \leftarrow \lambda N(s,a)$
  \ENDFOR
  \STATE $s \leftarrow s' ; a \leftarrow a'$
\UNTIL end of run
\end{algorithmic}
\end{algorithm}

We set $\gamma = 0.99$ and in each run computed the empirical future
discounted reward at each point in time.  As this value oscillated we
also ran a moving average through these values with a window length of
50.  Each run lasted for 50,000 time steps as this allowed us to see
at what level each learning algorithm topped out.  These results
appear in Figure~\ref{graph-3} and were averaged over 500 runs to get
accurate statistics.

Despite putting considerable effort into tuning the parameters of
Sarsa($\lambda$), we were unable to achieve a final future discounted
reward above 5.0.  The settings shown on the graph represent the best
final value we could achieve.  In comparison HLS($\lambda$) easily beat
this result at the end of the run, while being slightly slower than
Sarsa($\lambda$) at the start.  By setting $\lambda = 0.99$ we were able to
achieve the same performance as Sarsa($\lambda$) at the start of the run,
however the performance at the end of the run was then only slightly
better than Sarsa($\lambda$).  This combination of superior performance and
fewer parameters to tune suggest that the benefits of $\mbox{HL}(\lambda)$ carry
over into the reinforcement learning setting.

Another popular reinforcement learning algorithm is Watkins'
$\mbox{Q}(\lambda)$. Similar to Sarsa($\lambda$) above, we simply inserted the
$\mbox{HL}(\lambda)$ temporal difference update into the usual $\mbox{Q}(\lambda)$
algorithm in the obvious way.  We call this new algorithm
HLQ($\lambda$)(not shown).  The test environment was exactly the same as
we used with Sarsa($\lambda$) above.

The results this time were more competitive (these results are not
shown).  Nevertheless, despite spending a considerable amount of time
fine tuning the parameters of $\mbox{Q}(\lambda)$, we were unable to beat
$\mbox{HLQ}(\lambda)$.  As the performance advantage was relatively modest, the
main benefit of $\mbox{HLQ}(\lambda)$ was that it achieved this level of
performance without having to tune a learning rate.

\begin{figure}
\begin{minipage}[b]{0.49\textwidth}
\includegraphics[width=\textwidth,height=15em]{windworld.eps}
\caption{\label{windworld}[Windy Gridworld]
S marks the start state and G the goal state, at which
the agent jumps back to S with a reward of 1.}
\end{minipage}
\hspace{0.02\textwidth}
\begin{minipage}[b]{0.49\textwidth}
\includegraphics[width=1.07\textwidth]{graph-3-big.eps}
\caption{\label{graph-3}Sarsa($\lambda$) vs.\ HLS($\lambda$) in the Windy Gridworld.
Performance averaged over 500 runs.  On the graph, $e$ represents
the exploration parameter $\epsilon$, and $a$ the learning rate
$\alpha$.}
\end{minipage}
\end{figure}

\section{Conclusions}\label{sec:con}

We have derived a new equation for setting the learning rate in
temporal difference learning with eligibility traces.  The equation
replaces the free learning rate parameter $\alpha$, which is normally
experimentally tuned by hand.  In every setting tested, be it
stationary Markov chains, non-stationary Markov chains or
reinforcement learning, our new method produced superior results.

To further our theoretical understanding, the next step would be to
try to prove that the method converges to correct estimates.  This can
be done for $\mbox{TD}(\lambda)$ under certain assumptions on how the
learning rate decreases over time.  Hopefully, something similar can
be proven for our new method.  In terms of experimental results, it
would be interesting to try different types of reinforcement learning
problems and to more clearly identify where the ability to set the
learning rate differently for different state transition pairs helps
performance.  It would also be good to generalise the result to
episodic tasks.  Finally, just as we have successfully merged
$\mbox{HL}(\lambda)$ with Sarsa($\lambda$) and Watkins' $\mbox{Q}(\lambda)$, we would also like
to see if the same can be done with Peng's $\mbox{Q}(\lambda)$ \cite{Peng:96}, and
perhaps other reinforcement learning algorithms.

\paragraph{Acknowledgements.}
This research was funded by the Swiss NSF grant 200020-107616.

\bibliographystyle{alpha}
\begin{small}

redpajama_set_name: RedPajamaArXiv


Bethany Makawao Hair Salon. You have got to be kidding me. Bethany did a wonderful job on my hair my husband could not believe it. She is the most beautiful girl and more compationate about her job than any stylist I have ever been to. I definetly recommmed her. Those who disagree are living in the wrong era. Thank you Bethany!!
my review I have been going to Makawao Hair Saloon for quite a while, about 3 years, and never had any type of problem. I have always been totally satisfied with my hair cuts. Not only are they professionals, but a pleasure to do business with .
Bethany Makawao Hair Salon Bravo, Bethany. You did it again! Never, even once, have you had an off-day! I always get compliments on your haircut, and I cannot wait to return. Your finesse and expertise are unsurpassed. If you ever leave this country, you will never be replaced. I only hope Kevin as well as Holly appreciate you. You have style, Girl! In regard to Maestro, I need to ask Maestro in what? ...spelling error? ...offensiveness? Perhaps your haircut matches your self-esteem. I can send you a dictionary, for free, but it will not help your nastiness. Did you make a pass at her? You sound like you were rejected. Poor Maestro.
Yeah, Me Too! Yeah, Bethany! You did it again. My hair color is the best in the nation! You are so artistic!! This is a wonderful salon with a very gifted stylist--the best! I do not know how she does it; she is such a positive delight! Thank you, Kevin, for adding the best stylist ever to your salon! Bethany makes you feel so comfortable and does a lovely, lovely job on hair coloring which must be her specialty. Her cut has always been outstanding. I will never go to another stylist. You have my word. Lorence, I thought I saw you in the Rothschilds Most Negative Person in the World contest. I believe you won. I only hope when you shaved your head, you did not shave off your brain, too, funny girl. It sorta sounds like you did, however.


text:
Melania Trump's 'I really don't care' jacket sparks outrage from celebs
Nicholas Hautman
Jun 22nd 2018 12:41PM
Melania Trump had plenty of people talking when she wore a jacket with the words "I Really Don't Care, Do U?" scrawled on the back before meeting with immigrant children at the Texas border on Thursday, June 21.
The first lady, 48, wore the $39 green Zara military jacket as she boarded an airplane at Andrews Air Force Base in Maryland for an unannounced visit to Upbring New Hope Children's Shelter in McAllen, Texas. Though she changed into a cream-colored coat before going to the shelter, she put it back on when she returned to Washington, D.C., later in the day. Celebrities were quick to point out the unfortunate sartorial choice on social media and on TV.
"In the middle of the worst moral scandal in recent memory, so bad that her husband, [President Donald Trump], backed down for the first time, people who were supposedly on her side let her get on a plane with a jacket that said, 'I Really Don't Care, Do U?' For the record: We do," Stephen Colbert said on The Late Show With Stephen Colbert.
Melania Trump visits border detention facility
MCALLEN, TX - JUNE 21: U.S. first lady Melania Trump arrives at McAllen Miller International Airport to attend a round table discussion with doctors and social workers at the Upbring New Hope Childrens Center operated by Lutheran Social Services of the South and contracted with the Department of Health and Human Services June 21, 2018 in McAllen, Texas. The first lady traveled to Texas to see first hand the condition and treatment that migrant children taken from their families at the U.S.-Mexico border are receiving from the federal government. Following public outcry and criticism from members of his own party, President Donald Trump signed an executive order Wednesday to stop the separation of migrant children from their families, a practice the administration employed to deter illegal immigration at the border. (Photo by Chip Somodevilla/Getty Images)
US First Lady Melania Trump takes part in a roundtable discussion at Luthern Social Services of the South's Upbring New Hope Children Center in McAllen, Texas on June 21, 2018. - First Lady Melania Trump made a surprise visit to the US-Mexican border on Thursday, June 21, 2018 as her husband's administration seeks to quell a firestorm over migrant family separations. President Donald Trump first announced the trip by his wife, who will tour a non-profit social services center for migrant children, as well as a customs and border patrol processing center, according to a statement from her office. (Photo by MANDEL NGAN / AFP) (Photo credit should read MANDEL NGAN/AFP/Getty Images)
MCALLEN, TX - JUNE 21: U.S. first lady Melania Trump walks through the facility with program director Rogelio de la Cerda Jr. after a round table discussion with doctors and social workers at the Upbring New Hope Childrens Center operated by Lutheran Social Services of the South and contracted with the Department of Health and Human Services June 21, 2018 in McAllen, Texas. The first lady traveled to Texas to see first hand the condition and treatment that migrant children taken from their families at the U.S.-Mexico border are receiving from the federal government. Following public outcry and criticism from members of his own party, President Donald Trump signed an executive order Wednesday to stop the separation of migrant children from their families, a practice the administration employed to deter illegal immigration at the border. (Photo by Chip Somodevilla/Getty Images)
MCALLEN, TX - JUNE 21: U.S. first lady Melania Trump walks through the facility after a round table discussion with doctors and social workers at the Upbring New Hope Childrens Center operated by Lutheran Social Services of the South and contracted with the Department of Health and Human Services June 21, 2018 in McAllen, Texas. The first lady traveled to Texas to see first hand the condition and treatment that migrant children taken from their families at the U.S.-Mexico border are receiving from the federal government. Following public outcry and criticism from members of his own party, President Donald Trump signed an executive order Wednesday to stop the separation of migrant children from their families, a practice the administration employed to deter illegal immigration at the border. (Photo by Chip Somodevilla/Getty Images)
MCALLEN, TX - JUNE 21: U.S. first lady Melania Trump participates in a round table discussion with doctors and social workers at the Upbring New Hope Childrens Center operated by Lutheran Social Services of the South and contracted with the Department of Health and Human Services June 21, 2018 in McAllen, Texas. The first lady traveled to Texas to see first hand the condition and treatment that migrant children taken from their families at the U.S.-Mexico border are receiving from the federal government. Following public outcry and criticism from members of his own party, President Donald Trump signed an executive order Wednesday to stop the separation of migrant children from their families, a practice the administration employed to deter illegal immigration at the border. (Photo by Chip Somodevilla/Getty Images)
Jimmy Fallon said during his monologue on The Tonight Show Starring Jimmy Fallon, "Nobody really knows what it means or if it means anything at all. Meanwhile, Trump's going through his closet like, 'Where is it? Where's my coat?'"
Over on Twitter, Jane Lynch wrote, "Is @FLOTUS so unfamiliar with English that she doesn't know what the back of her jacket today says? That her initiative #BeBest is grammatically incorrect? Is there NO ONE ON HER STAFF to help her with these things?"
Is @FLOTUS so unfamiliar with English that she doesn't know what the back of her jacket today says? That her initiative #BeBest is grammatically incorrect? Is there NO ONE ON HER STAFF to help her with these things?
— Jane Lynch (@janemarielynch) June 21, 2018
Chelsea Handler echoed, "The hypocrisy of @FLOTUS going down to the border to speak broken English to our refugees in that dumb jacket is a little hard to swallow. For all the trump supporters who defend her by saying she speaks five languages—English isn't one of them."
The hypocrisy of @FLOTUS going down to the border to speak broken English to our refugees in that dumb jacket is a little hard to swallow. For all the trump supporters who defend her by saying she speaks five languages—English isn't one of them.
— Chelsea Handler (@chelseahandler) June 21, 2018
Patricia Arquette tweeted, "Melania Trump just won worst dressed for the century." Julia Louis-Dreyfus wrote, "It's endless with these morons."
Melania Trump just won worst dressed for the century.
— Patricia Arquette (@PattyArquette) June 21, 2018
It's endless with these morons.https://t.co/Zuph1E6qw7
— Julia Louis-Dreyfus (@OfficialJLD) June 22, 2018
The former model's press secretary Stephanie Grisham responded to the backlash in a statement to Us Weekly on Thursday afternoon. "It's a jacket. There was no hidden message," she said. "After today's important visit to Texas, I hope the media isn't going to choose to focus on her wardrobe (much like her high heels last year [in Texas while helping Hurricane Harvey survivors])."
The president, 72, later claimed on Twitter, "'I REALLY DON'T CARE, DO U?' written on the back of Melania's jacket, refers to the Fake News Media. Melania has learned how dishonest they are, and she truly no longer cares!"
"I REALLY DON'T CARE, DO U?" written on the back of Melania's jacket, refers to the Fake News Media. Melania has learned how dishonest they are, and she truly no longer cares!
See more stars' reactions below.
No. This is correct. There is no hidden message. The message is right there for everyone to read clearly. https://t.co/dgAiqCk0uZ
— Busy Philipps (@BusyPhilipps) June 21, 2018
No message? The jacket literally has a message on it. How does no one running this country understand the meaning of optics. Jesus! https://t.co/thBD9XxxCb
— Jesse Tyler Ferguson (@jessetyler) June 21, 2018
Melania in her"I don't care jacket" JUST"HAPPENED"TO SHOW UP UNANNOUNCED..TO DOCTORS,
SOCIAL WORKERS,&MEDICAL STAFF.
WERE THEY THE SAME STAFF WHO LOCKED CRYING TODDLERS IN CAGES⁉THE SAME STAFF WHO MADE THEM SLEEP ON FLOORS WITH TINFOIL BLANKETS⁉ https://t.co/kmobz35E0C
— Cher (@cher) June 21, 2018
Is this for real? https://t.co/17tDIi35Ou
— Katie Couric (@katiecouric) June 21, 2018
pic.twitter.com/GmrR5p3BAh
— Zendaya (@Zendaya) June 21, 2018
Is this a joke? I didn't think anyone could be that out of touch with reality... https://t.co/F9MowJDPnF
— Lili Reinhart (@lilireinhart) June 21, 2018
Wait, wait, wait. I just got wifi to work on my flight. This Melania jacket thing cannot be real. Can't. I'm choosing to not believe it.
— Chris Hayes (@chrislhayes) June 21, 2018
More from Us Magazine:
Selma Blair celebrates two years of sobriety: 'I am a living miracle'
'RHOC' alum Alexis Bellino splits from husband after 13 years of marriage
Seth Rogen trolls Ariana and Pete's social media flirting
flotus
stars react
This Man Went To Help A Lonely Dog At A Gas Station. Then He Looked Closer At Its Tag
Surprising Secrets About Flying That Will Change The Way You Fly
'80s And '90s Stars Who Turned To Crime
Little-Known Facts About The First Families
Taylor Swift reveals surprising inspiration behind new song: 'I was really upset'
Kelly Ripa canceled daughter Lola's debit card for this relatable reason
AOL Logo Entertainment


Marketing, behavioural economics and the fuzzy grey area where the two meet.
Angry. Seething. Incensed with a side of outraged, and a generous sprinkling of righteous indignation on top.
Fringe Theatre Adventures' decision to reject Daniel Hughes as a volunteer, with a form letter, and an icky token offer of, well, a token, was appalling, unforgivable and revealed a fundamental gap in their understanding and valuing of volunteers humans.
It showed a disturbing able-ist mindset within the organization, one which might not be shared by everybody connected, but by enough people that it influences policy.
And since the Fringe Festival gets thousands of unpaid hours every year from their volunteers, the philosophy of "treat them better than gold" should not be rocket science.
Apparently for some it is. Whoever was at the helm of their social media doubled down with the contradictory statement, "The Fringe is an inclusive organization that provides many opportunities for people with disabilities to volunteer," ironically after conclusively proving that they are not.
This was followed by one of the worst, most diminishing statements I have ever seen by Executive Director Jill Roszell (and as a communications professional, I actually study this stuff).
The community came unglued. (I am still looking for some crazy glue to reassemble myself.) And rightfully so. This was a horrible situation that was terribly handled.
For the record, it is a great example of what they should have said in the first place. The Fringe took full responsibility for their mistakes, weren't defensive in the least, and promised to not only fix the problem but keep the public informed as to how exactly they are going to fix it.
Whether they accomplish that goal remains to be seen. And we, as the public, need to keep their feet to the fire to make sure they do so. We owe it to ourselves, and we owe it to the entire community they completely diminished with their actions.
They have a real opportunity to make a change for the better here. Let's give them a little breathing room to do it, but not let up on the scrutiny, or our expectations that they make this right.
And they may fail. In fact, even if they recover, they will still likely bear the brunt of this mistake at this year's festival, and perhaps they should.
But please, please don't punish the artists.
Having performed at the Fringe many times over the last 20 years, there is one thing that has always bothered me about the setup of the festival. There is nothing to be done about it, and it is patently unfair, but that's just the way it is: The artists have no leverage over the festival.
Even if you breathe fire.
There has been a lot of talk about boycotting the Fringe this year. And though I understand the urge to do that (see: "Seething anger. Seething." above) it's important to understand that the effect of that action is going to be relatively insignificant on the people who made the mistake, and devastating to an entire community of artists who had nothing to do with it.
Artists are dependent upon the Fringe, not the other way around. For each ticket you refuse to buy, you're withholding a couple dollars from the Fringe - a large organization with corporate sponsors - and over $10 from artists who earn close to the poverty line, and are counting on this festival for a much needed chunk of change as part of their annual income. In this instance, punishing the artists does nothing but punish the artists.
It's kind of like kicking your dog to punish your son for pulling his sister's hair.
So what can we do? How do we make our voices heard? How do we constructively vent our outrage?
Make sure the Fringe knows what the core of the problem is. Letters, phone calls, social media, etc. Don't just rage at them, but offer genuine feedback. Show them what they did wrong and how to fix it. Although it is absolutely no excuse, I fear that the people who made these decisions really didn't know better. That's seriously problematic, but this is an opportunity for the Fringe to really up their game.
Give them some time to come up with a very visible response that actually speaks to the core of the problems. They promised it, let's give them the opportunity to deliver.
Don't buy programs. It's all online anyway, so you're not necessarily missing out.
Don't go to the beer tents. The Fringe makes a noteworthy chunk of change there. The local businesses would probably be thrilled to sell some more pints.
Don't buy merchandise. If they continue to make mistakes like this, do you really want a keepsake reminder of the year the Fringe went bad?
Let their sponsors know, as visibly as possible, what you think. Make sure it's clear to them that you see their businesses in a poor light for supporting a festival with this policy. The Fringe losing a sponsor over this would be a massive shakeup.
But first, again, I strongly suggest giving them time to fix it. Not letting it go, but saying, "Alright, you say you want to do better. Show us." And holding them to it.
The Fringe has an opportunity to learn from this. To fly or fail. This could be a chance for the Fringe to learn what inclusiveness actually means. And if they fix it, go, enjoy the beer tents, buy merchandise, drop donations at the gate. Send a letter to the senior executives of a sponsor to say how impressed you are. Reward the Fringe for making it right.
But whatever happens, please don't punish the artists. See some shows, drop some bills in a street performer's hat. It wasn't their fault, they don't have any power in the situation. The last thing we want to do is punish the wrong people.
Isn't that exactly what we're angry at the Fringe for doing?
UPDATE: Daniel and his family have also asked people to not boycott the Fringe. If we are upset on his behalf, we owe it to him to respect his wishes, don't we?
I kind of feel like this situation is getting blown somewhat out of proportion.
Maybe the fringe could have told Daniel in a better fashion than they did. But are we really going to vilify the fringe for wanting people who can perform the tasks they're required to perform in any given volunteer position?
Yes we are going to vilify the Fringe because they are acting like villains.
This young man has performed the duties at previous Fringe Festivals. The whole concept of the festival is embracing those on the "fringe" of society. But here they are excluding those that they are supposed to represent. There are other options such as pairing Daniel or others like him with a partner volunteer but that doesn't even seem to have been considered. Failing to cite real reasons that rational people could understand such as: it creates an unreasonable burden (with examples), costs too much money (with justification) or that there are insurance issues (if there are any) would have made this more palatable and professional.
The thing is, their communication with Daniel shows a deep-seated lack of values when it comes to dealing with volunteers, and a very able-ist slant. So much so that they sent out form letters, and thought they could make things right with a token gesture at a 'community party.' This goes against any claim to being an inclusive festival, and shows a disrespect for their volunteers - of any ability. There are many festivals in this city who have been using volunteers of all ability levels for years and not followed as disrespectful a course of action as this, so there's not a lot of excuse. Their original responses to the controversy also reinforced their values.
If we want to live in a better world, this is exactly the thing we need to call people and organizations out on.
And I'd say Daniel, as one of the volunteers who was directly affected by this, would not say things are being blown out of proportion.
Seems like a bunch of social justice warriors crying over spilled milk. So the disabled dude can't do the job the festival wants him to do, that's how it goes.
Daniel has volunteered for many years and is suddenly not able to perform duties that he has been volunteering for in the past? There was no consideration of alternatives (helping to clean the venue, simply walking around with a volunteer shirt on and smiling at people etc) This is an unpaid and thankless position and the organizers should be grateful for anyone - regardless of "ability" who is willing to give up time.
© 2010 Randy Brososky. Watermark theme. Powered by Blogger.