text:
Category: employees
'What To Learn About Change Communication From Walking Dead'
Change Communications, Employee Communications, employees, The Walking Dead
change communications, Rick Grimes, The Walking Dead, zombie apocalypse
What To Learn About Change Communication From The Walking Dead. There are millions of people who watch AMC's The Walking Dead. In case you do not here is the story in a nutshell; the program is based on a graphic novel series about a small group of people with nothing in common other than they survived a global pandemic and want to remain alive. They are victimzed by zombies (known as 'walkers' in the series) and gangs of other survivors.
Lately the group was accepted into a walled community south of Washington D.C. called Alexandria. The veterans of this sustainable and upwardly mobile community have been spared the grief and violence the new world has because of the location and enormous and well constructed walls.
'What To Learn About Change Communication From Walking Dead. Expect there will be resistance to new ideas and change in the workplace. Also recognize that it will not be as bad as it is on The Walking Dead. In the photo Rick Grimes delivers change communications to the people of Alexandria.
Rick Grimes, a former law enforcement official, is a veteran survivor and has seen the worst of the post apocolytic world. He believes (rightly) that the inhabitants of Alexandria are living on borrowed time. That they will be killed by 'walkers' (who swarm by the thousands) or by one of the other violent, roving gangs of survivors. He wants to teach the people of Alexandria how to make it in the real world. With that, here is the connection to change communications and what the rest of us can learn.
New people are suspect. Grimes and his group are only accepted into Alexandria after passing a battery of interviews and the approval of Alexandria's leadership. They are housed together, not allowed to have weapons or leave the grounds. New is different, hard to get used to, and suspicious. New means change and no one likes to change whether in the office, the plant, the shop floor or Alexandria.
New ideas are usually rejected. If you work in change communications or just try to have a new idea you have already heard, 'that's not the way we do it here' or 'why do we need to change, everything was going great until you showed up.' Grimes wants weapons training and regular patrols to watch for walkers or looters. If you have worked in change communication and been outnumbered in the workplace, you know how isolating this is. Even with the support of company leadership (which Rick Grimes has too) you know that when those leaders are out of site it is very difficult to get the troops to even consider doing something new, nevermind adopting it. Very few support Grimes ideas.
Passive aggressive resistance. The people of Alexandria pretend to be interested in Grimes' ideas, but they do not pretend very hard. In fact, behind his back several conspire to try and kill him. The leader of the passive aggressive resistance is the local doctor (who abuses his wife and children and Grimes has a thing for the wife, but let's not get too deep into that) who openly opposses Grimes. In the end, Grimes kills the doctor which for the workplace is analogous to firing someone. Grimes tried to work things out with the doctor and change leaders will try with the passive aggressive people too. But the lesson for the workplace is that there are times when firing someone is the only thing to do. There are plenty of people who will convince themselves and anyone else who will listen that theirs is a righteous cause. Horsefeathers. The unknown and adapting to change is scary and uncomfortable. No one wants to be scared or uncomfortable. While a rational person would try to adapt remember that humans are not rational. The rational part of the human brain is the newest and least evolved. The emotional parts are far more evolved and that is why people react emotionally. It will not end with the firing/killing as those people will always have followers. Instead of seeing the logic of 'addition by subtraction' people will get mad and seek revenge.
Stay the course. In spite of the hostility directed at him, Grimes continues to teach the hapless Alexandrians how to survive, and takes direct action in the fight to outlast the zombie hoard. In the office, do like Grimes does. Stay focused on your goals. Be approachable and offer to teach the willing but do not negotiate. Negotiations in change communications will open the door to more and more exceptions. You do not want that and the people who hired you do not either. Remain unphased by whiners, moaners and complainers. Reward those who adapt and change. And make both firing and rewarding very visible.
It is not enough to be right. Remember that in the resistance to change movement with the attendant emotion attached, being right will not matter. In fact direct challenges often lead to more resistance. People double down on the ideas they have held for so long. The Alexandrians are starting to see the logic of what Grimes is trying to accomplish for them. But this has taken several episodes. Take the lesson, stay the course and eventually the rest of the group will catch one. Just give them time.
To be in touch, send an email to haroldnicoll@gmail.com.


text:
We use cookies to make this service simpler. Find out more about cookies
UK Government and Parliament
This petition was submitted during the 2010–2015 Conservative – Liberal Democrat coalition government
View other petitions from this government
Petition Return VAT on Air Ambulance fuel payments.
The Air Ambulance Service is forced to meet rising fuel prices year on year including VAT.
The Air Ambulance Service have saved successive governments millions and millions of pounds funded by charitable donations given by the general public to run what has proven to be an essential service.
Whilst the Lifeboat Service has been exempt from VAT on fuel costs since 1977, a similar privilege has not been afforded to the Air Ambulance Service;
We call on the government to have an urgent review of this situation and in doing so
We call on the government to return in the form of grants to Air Ambulance Service providers all the future VAT which the Treasury collects from them so that the Air Ambulance Service is in practice exempt from paying VAT in the same way as the Lifeboat Service.
This petition is closed This petition ran for 6 months
154,662 signatures
Parliament will consider this for a debate
Parliament considers all petitions that get more than 100,000 signatures for a debate
Waiting for 3,887 days for a debate date
Government responded
This response was given on 22 March 2012
The Leader of the House of Commons has written to the Backbench Business Committee, who are responsible for the scheduling of debates on e-petitions, informing them that the petition has reached 100 000 signatures
The Backbench Business Committee meets weekly to hear representations from MPs for debates in backbench time. The Committee can consider any subject for debate, including those raised in e-petitions, but an MP must make the case for their consideration. More information about the Committee is available on its website http://www.parliament.uk/bbcom
The Government has provided the following response to the petition:
"Charities are at the heart of the Government's ambition to build the Big Society. To help support this, the Government provides charities with over £3 billion a year in tax reliefs, which includes around £200 million in reliefs from VAT. Furthermore, as part of the wide range of VAT and other tax reliefs available to charities, voluntary search and rescue charities can purchase medicines, medical equipment and certain other specialised equipment VAT free. All of these reliefs are derogations from the normal EU VAT rules and are not enjoyed by charities in other Member States.
Air ambulance charities currently operate in many different ways, which will determine whether they pay VAT on supplies received, and the extent to which this can be recovered. There are a number of specific VAT reliefs which apply to air ambulance charities. Full VAT relief applies to the purchase of air ambulance helicopters, saving £600,000 on the cost of a £3 million helicopter, and on leasing costs, saving £86,000 a year based on a standing charge of £36,000 a month. If the lease contact includes fuel, then this will also be zero rated. Furthermore, aviation fuel can be purchased exempt from excise duties, and at a reduced VAT rate of 5 per cent.
However, there are some charities that do pay the standard rate of VAT on fuel and it is right to look further at whether the impact of VAT can be mitigated. The Government fully appreciates the valuable work carried out by all rescue teams, and is committed to reviewing tax and operational issues around the Air Ambulance Service. It is important that we fully understand the operating models and funding arrangements of air ambulance services before narrowing our focus onto the specific issue of VAT.
The Department of Health is committed to undertaking a 'fair playing field review' which will examine whether there are inequalities in the treatment providers of NHS funded services, and if so what can be done to address these. In addition they are considering the capability and funding of emergency medical care of the type provided by the London Air Ambulance. HM Treasury officials will be working with colleagues in the Department of Health to discuss both reviews. If neither addresses the issue of the VAT regime applicable to the air ambulance service, the Treasury will undertake its own separate review."
Guy Opperman MP and Hugh Bayley MP were allocated a half-day in the Chamber in Backbench time and the debate took place on 11 July 2012. A transcript of the debate can be found at http://www.publications.parliament.uk/pa/cm201213/cmhansrd/cm120711/debtext/120711-0003.htm#12071139000002
Date closed 10 February 2013
Get petition data (json format) About petition data
The data shows the number of people who have signed the petition by country as well as in the constituency of each Member of Parliament. This data is available for all petitions on the site. It is not a list of people who have signed the petition. The only name that is shared on the site is that of the petition creator.
How petitions work

redpajama_set_name: RedPajamaCommonCrawl


text:
How Gaming Helps Me Get Over My Fear Of Flying
Holden Miller
Published 9 years ago: January 31, 2013 at 1:00 pm -
Filed to:feature
ghost trick phantom detectiveipadkotakucorekotakumobileopinionosmosthe world ends with you
My Apple iPad 2. Crisp display with a density of 132 ppi. Reacts to multi-touch gestures. Contains accelerometer, gyroscope. Reacts to involuntary shakes in hands. Enough battery life to last majority of a cross country flight. Reacts poorly to excessive, clammy sweating. Does not register uncontrolled clenching of fists/jaw. Eight-megapixel front-facing camera accurately captures wild desperation in eyes, ghostly pallor, but leaves out racing thoughts, mounting dread, cavalcade of head voices.
Plays games. Games help. Flying sucks. But it sucks more without approved electronic devices.
I am definitely not alone in my fear of flying. This common fear affects up to one-third of Americans, and I think the fact that it's so widespread has led to a kind of dismissive attitude towards it. Either it's treated as a joke, or as something that only pills can fix. Practical ways of dealing with fear of flying get short shrift. In the scheme of things there are much worse problems, but that doesn't make it insignificant. Flying today is unavoidable, and can be a huge source of stress.
It's certainly not a panacea, but recently I've come to find that playing games on my iPad can alleviate my fear of flying more than watching movies, certainly more than reading a book, and somehow even more than my beloved comic books. I know that trying to be prescriptive about phobias is difficult, and what's often most helpful for me is just hearing about someone else's experience. So in that spirit, I'd like to talk about how games help.
Why I'm Afraid
Several years ago on a flight from Idaho into California the plane I was on encountered an unexpected coastal storm. We were preparing to land, and at relatively low altitude. In the small plane the turbulence was very violent. Passengers were pretty distressed, myself included, and only more so when alarms started sounding from the cockpit, including a pleasant female robo-voice repeating "wind shear, wind shear."
You don't want to see me when I fly. Shaking, sweating, crying: it's not pretty.
Wind shear, I later learned, is abrupt and dramatic shift in wind speed and direction. Imagine a strong wind blowing at your face only to suddenly be pushing you forward. Weirdly enough, it's not good for flying. I totally lost my cool and started furtively texting friends and family that I loved them — yes, my phone was on. Thankfully, after bouncing around and then plummeting towards the ground for a little while, the plane was able to recover enough to divert to another airport. Believe it or not, we encountered the same problem at the second landing, site which led to another round of panicked out-loud cabin prayers and a final diversion to LAX.
And that's it. No injuries, no in-cabin fire or water landing. Millions of people have gone through the same thing, but for some reason I couldn't shrug it off. Hell, when we were finally landing at LAX I could see at least a dozen planes flying parallel to us, refugees from other airstrips. We landed, the pilot was kind enough to announce that he had been flying since Vietnam and it was the worst landing he'd ever done, and we got off the plane. I had to fly back across the country four days later. It wasn't easy anymore.
You don't want to see me when I fly. Shaking, sweating, crying: it's not pretty. The fear is slowly improving, but I still get intense anxiety while flying, and up to a week or more before and afterwards. I dream about it. Frequently. When I see planes flying above me, I get scared. Even watching people fly in movies or TV puts me on edge.
This is all, of course, before I get on the plane. Once there not only do I scare myself — half to death, my heart tells me — but also the poor people who have to sit beside me. I've tried a lot of things to help it, chemicals included. But it's that precious, wonderful time when you're sealed for hours in a tube miles above ground/ocean/comfort that help is desperately needed and surprisingly spare.
Games Put You In Control
I had tried playing my 3DS while flying in the past, but it didn't work for several reasons. 3D was totally out of the question, and concentrating on the small screen made me prohibitively self-conscious about what I was doing. But mostly it was the DS's heavy reliance on stylus: one errant altitude drop and I'd end up stress-jabbing that bad boy right into my thigh. I don't own a Vita so I'll reserve judgement there. It wasn't until I inherited a friend's old iPad that I found a way to game in midair.
The first therapeutic experience I had with a game while flying was Capcom's Ghost Trick: Phantom Detective. An excellent game in any setting, I turned to the it on a cross-country jaunt instead of A) watching Ted, B) dragging my fingernails across my pant legs repeatedly, C) hyperventilating in the bathroom, or D) chugging tiny bottles of hooch.
TIPS FOR FLIERS – What gamers can do when they're feeling afraid.
DO: Play a game that will engage you on multiple fronts, and remember to crank up your headphones.
DO: Play a game that puts a little fun in death — no The Walking Dead.
DO: Play a game you're already good at. Feel like a winner again!
DO: Play a game with a strong but lighthearted story. Nothing says distraction like compelling narrative.
DON'T: Play anything really frustrating. If you're peeved on the ground, you'll be apoplectic up there.
In Ghost Trick you control Sissel, a mysterious man with wonderful hair, who's recently been relieved of his life and is trying to solve his own murder using things called "Ghost Tricks." Sissel's spirit can go back in time and possess objects, manipulating them to change his future. Playing, I felt the surety of my own demise slide a little further away. "What are these beautiful magicks?!" I thought to myself.
In a game good for flying, death is reversible, avoidable, and altogether Not That Bad. Of course, most games feature a "back from the dead" mechanic, but it helps if your character doesn't just reappear a lá Mario. After all, who's that new Mario? Is he the same? Did he retain the sentience of the previous one? If Mario was on a plane and it crashed, would he just pop out of a warp pipe back at the departure gate? Existentially, it's a nightmare. Ghost Trick, on the other hand, has death and life a mere button click away, your character's consciousness sliding seamlessly between the two. Even better, the world of the dead is packed with colourful characters like a talking lamp and a cuddly dog named Missile who — hey we're already past Des Moines? All right!
Best of all, playing it allowed me to feel like I was in charge. A major contributor to my fear of flying is feeling that I have no control, especially concerning death. Not that I have it otherwise, but no one said this was rational. After an hour of Ghost Trick, every time the plane made a noise and sent me into a frenzy, I would imagine how, like Sissel, I could "trick" my way over to the offending part and repair it in a jiffy. This fantasy was made all the easier because over the past two years I've taken an unhealthy amount of time familiarising myself with aircraft construction, aerodynamics, and the most frequent contributors to… incidents. So I knew (or at least could fudge) exactly what bolt, flap, or gauge needed tricking. Ridiculous, yes, but an amount of levity that can be hard to come by in time of panic. Speaking of levity…
Games Make The Afterlife Seem Fun
Replace your flying fears with fashion in Square Enix's The World Ends With You! The JRPG was originally released in 2008 on the DS, but I prefer the newer touch version. The touch version's deep-but-swift combat is just hard enough to be distracting, but intuitive and malleable enough to not be frustrating. As in Ghost Trick, the protagonist Neku is trapped in an afterlife. This one is a funky high-fashion Shibuya. He's participating in the "Reaper's Game," a sort of seven-day alternate reality game, to win his life back.
On a flight I noticed not only did it have the aforementioned agency angle, but it also made my initial fear look like a dynamic world of fun and friends. The world of the game is so cool that one of the main characters even volunteers himself for it. I am not religious, at all. I do not believe in an afterlife. But I don't dismiss the adage that everyone finds god in a foxhole. Any lie, any fantasy you can sell to yourself is a boon.
So for the duration of a flight, my harried brain had no trouble adopting the belief that engine failure would leave my somewhere exciting (as a diehard LOST fan, I've indulged similar fantasies in the past). Fully functional, tres exciting, and full of Important Lessons, the afterlife of TWEWY is the place to be. It's chock full of secret ramens, haute couture, maths puns (so many maths puns…), and crazy as hell music. Music so crazy and incessant you might just get hypnotized. Distracted, even.
Games Engage Multiple Senses
One thing about fear is you have to pay attention to it. Rarely is it a feeling simmering in the background, instead preferring to vice grip your brain.Trains of thought are quickly interrupted by the base, brute, repetitive "you are not safe. You will not leave this plane. You will perish in this outfit. No one here knows who you are. Why in the world did you board this thing. You poor, dead idiot." Rolls off the tongue.
Playing games on my iPad, especially with the headphones turned up, works to pry a few of those five fingers from around my grey matter, sense by sense. The best example I can think of here is Osmos, the powerfully soothing PC game which made its way onto touch platforms some years ago. In Osmos, you control a sphere through tranquil environments, consuming smaller spheres and dodging currents, obstacles, and more. It's like a bare bones Katamari on ludes. I appreciated Osmos before, but I never really loved it until I played it whilst flying.
The almost synesthetic mix of sound, visuals, and touch was able to elbow out some of my fear for sheer lack of real estate. Cloud front: I didn't see you. Whistling noise: I didn't hear you. Turbulence: what turbulence? It was a proactive approach, attacking the problem on multiple fronts. Quite literally, I couldn't yank out my own hair if I was too busy beautifully absorbing smaller motes and drifting away to flowing ambient music. Osmos: you won't have a heart attack!
Games Make You Feel Competent
An aspect of my fear of flying that I never would have predicted is the embarrassment involved. When I'm flying alone, I often spend several hours before a flight walking around the airport, dragging my bag behind me because I'm too nervous to sit. True or not, I think by the end some of the restaurant hosts begin to recognise me. Onboard is no better. When I look around the plane, everyone looks so placid that quite frankly I feel like a loser. All I have to do is sit down: how can I be so bad at it? All of these games and more can hand you back some of your dignity through small victories. The fear doesn't totally neuter you, you can still put up a high score damnit!
A little digital pat on the back can be just the thing when the girl next to you is whispering to her boyfriend in the aisle seat that you're acting weird. When that guy might be judging your jelly legs as you walk down the aisle. When your seat mate is aggravated at you for taking up the window space because staring straight through it helps just that much. I don't mean to be strange or annoying. Regaining my self-esteem makes me feel less small, and more in charge of my own emotions. You get to win. It feels nice.
Just Avoid The Frustrating Ones
You'll notice I've left out mobile gaming's bread and butter, bite sized puzzlers. I've played as much Where's My Water? as the next guy who plays a lot of Where's My Water? (and may own a Swampy plush hey who said that). At home or on the subway, I'll gladly spend a long time trying to direct that stream of water into a subterranean crocodile's bathtub, collecting all the rubber ducks along the way. But when that "fasten seatbelt" sign flashes on, the victory of 3 stars seems mighty pyrrhic.
What at sea level would be frustration expands at high altitude into self-loathing and acknowledgement of the pointlessness of the earth. Because of course that bird would stop rolling one pixel away from the final pig. What else would happen? Now let's look out the window until a pitot tube malfunctions and inexorably alters the aircraft's angle of attack, leading to aerodynamic stall. Zero stars. Best to stick to story-driven games, or at least a puzzler that puts tranquility above difficulty. Otherwise you just might end up freaking out. Even more.
I don't want to be afraid on planes, and I certainly don't like it. But I also can't pretend it's something that's just going to go away on its own. I spent enough time doing that. Discovering a way to at least partially fight a phobia can feel like a light at the end of the tunnel, however dim.
Nec says:
Nah no reason to be afraid of flying. If you have to fly and something bad happens you're doomed anyway so why stress about it, enjoy the flight and if they suddenly tell you to put your head between your knees, just remember to kiss your ass goodbye while you're there.
yourname says:
TAHTS WEY THE TERARS FO PHLIN
HH says:
Don't be afraid of dying because you will die?
Nimhain says:
While I'm not afraid of flying, I'm not the most comfortable flyer. I get cabin fever/restless when stuck on long flights. I've found playing games has always helped me a lot. I'm particularly fond of saving Professor Layton games until me next trip over (though with only a couple of titles left for me to play, will have to find something new).


The Mountain Bike Guide - online shop selling maps and cycling guides for The Peak District and Derbyshire.
The maps you will need to cover the rides in my book are available online from Amazon.co.uk. To purchase any item shown below just click on the picture or the 'Buy from Amazon' link. This will open a new window showing that item on Amazon's web site.
This map covers most of the routes in the southern part of the area. This and Dark Peak are the two most important maps if you want to enjoy the full range of rides described in the book.
This map covers most of the routes in the northern part of the area. This and White Peak are the two most important maps if you want to enjoy the full range of rides described in the book.
This covers the area just to the South of the White Peak map and is needed for the Belper ride.
This covers the area south and west of Derby and is needed for the Netherseal - 4 Counties ride.


Heslop, J A; Hammond, T G; Santeramo, I; Tort Piella, A; Hopp, I; Zhou, J; Baty, R; Graziano, E I; Proto Marco, B; Caron, A; Skold, P; Andrews, P W; Baxter, M A; Hay, D C; Hamdam, J; Sharpe, M E; Patel, S; Jones, D R; Reinhardt, J; Danen, E H J; Ben-David, U; Stacey, G; Bjorquist, P; Piner, J; Mills, J; Rowe, C; Pellegrini, G; Sethu, S; Antoine, D J; Cross, M J; Murray, P; Williams, D P; Kitteringham, N R; Goldring, C E P; Park, B K (2015). Concise review: workshop review: understanding and assessing the risks of stem cell-based therapies. Stem Cells Translational Medicine, 4(4):389-400.
Rigter, H; Pelc, I; Tossmann, P; Phan, O; Grichting, Esther; Hendriks, V; Rowe, C (2010). INCANT: a transnational randomized trial of Multidimensional Family therapy versus treatment as usual for adolescents with cannabis use disorder. BMC Psychiatry, 10:28.
This list was generated on Mon Apr 22 10:13:26 2019 CEST.


text:
November 8, 2018 April 26, 2019 Morag Lee USA twin-break
The USA and Canada are both amazing destinations in their own right and bursting with incredible cities, mountainous national parks, and fascinating history. But between them, they cover a 19.819 million km squared area (larger than the world's largest country: Russia) so there's a lot of ground to cover.
To make it easier, keen travellers have the option of Canada/USA twin-breaks thanks to the two country's train, road, cruise and air network. While theoretically you can combine any mishmash of cities and towns into a USA/Canada twin-break, there are some destinations that are more popular and logistically simpler than others. Here are just five of the easiest USA cities to fit into a Canada adventure.
The dazzling sights of NYC should be on everyone's bucket list. Especially when you consider that it's only five hours from the UK and is a popular starting port for New England and Atlantic Canada cruises (along with New England's other coastal destinations like Rhode Island and Boston).
Also, the Niagara Falls lie along the border between the province of Ontario and New York State. While it is a six-hour drive between NYC and Niagara Falls, many travellers do make the journey and come back singing its praises.
2. Seattle
Sitting only 135 miles below the USA-Canada border is the city of Seattle. If you're looking at a self-drive, then you can drive from Seattle to Vancouver via Interstate 5, crossing the border at Blaine, Whatcom County.
Seattle, however, is also a popular cruise port for Western Canada and Alaska cruises. In fact, most cruises that start or stop in Seattle will sail further north to Canada and Alaska.
It might be over 800 miles south of Seattle, but cruises that start in California and travel north to Alaska are not uncommon. Right now we're offering an itinerary on-board the Grand Princess that starts in San Francisco and stops by Juneau, Skagway, Glacier Bay, Ketchikan, and Victoria – and includes a journey along the Inside Passage.
4. Las Vegas
Not all Canada/USA twin centres involve a cruise line. Las Vegas is nowhere near a coastline and is over a thousand miles from the Canadian border, but there are regular flights between the two cities and many travellers do make the journey between these two fabulous cities.
5. Anchorage and Juneau
This one is a bit obvious, but Alaska as a whole is very easy to combine with a Vancouver holiday. To start with, there's a generous selection of cruises that start in Vancouver and then make the journey along the Alaska coast. But you can also combine an Alaska adventure with a scenic tour through Canada's northern province of Yukon.
Whitehorse – Yukon's capital – is nine hours between both Anchorage and Juneau (Anchorage being to the west, and Juneau to the south). Both roads pass through other scenic towns and national parks (including Skagway and Wrangell-St. Elias National Park & Preserve) to break up the journey.
A food and drink guide to Victoria


A recent road rage incident in the parking lot of a Jackson, MS Walmart has landed one person behind bars.
According to the report I read, one person gave this driver the car, but when the original owner of the car asked for it back---this happened.
The driver of the car can be seen deliberately ramming the car into the concrete posts outside the store.
Just as police arrived on the scene, a man walks up to the driver's side window and punches it out in an attempt to stop the driver.
Luckily no one was injured here and the driver was arrested by police.
Now we know why stores have these type posts in front of them. To protect the store in crazy incidents like this.


Making it in Hollywood was never easy. Many famous and now huge stars had some pretty tough times becoming what they are and who they are today.
Numerous celebrities had to wait tables, wash dishes or work in grocery stores just to make ends meet and have a place to stay.
Some didn't even have that and lived on the street not knowing what tomorrow holds, hoping for the best.
For others, waiting tables was too mundane and the money simply didn't come quickly enough, thus they decided to turn to some drastic measures.
Adult movies were a sure thing and easy money. If you were decent looking and, more importantly, willing to take your clothes off in front of camera, that precious dime was just around the corner.
Ok, this was too easy. For someone named Matt LeBlanc, beginning a Hollywood career in adult films was practically written among the stars. He probably didn't even have to change his name for the role since LeBlanc was already dirty enough.
Ok, now we know where the famous "How you doin'?" pick up line comes from.


The Beaufort Natural Range, an exciting collection crafted from natural oak with a light lacquer to express the natural grain of the wood. This range is the perfect addition to a classic traditional home. An everlasting style.


NENA 2017 is the can't-miss show of the year for 9-1-1 professionals. The exhibit floor will be filled with thousands of public safety professionals, telecommunications specialists and policy makers. NENA 2017 will provide the opportunity for exhibitors to network and interact with public safety decision makers from all over the US and abroad. NENA is extremely proud of the growth that the annual conference and expo has undergone since 2009. NENA is dedicated to providing exhibitors and attendees a world-class experience. NENA works hard to attract more decision makers to your booth through extensive pre-show promotion, on-side activities, social media marketing and more. If your business touches the public safety communications marketplace your business should be exhibiting at NENA 2017. Book your space today!


text:
\section{Introduction}

Universal approximation results for feedforward neural networks \cite{Haykin2009a} state that certain function classes can be approximated to any accuracy by large networks of simple compositions of simple activation functions. The survey \cite{Pinkus1999a} reviews seminal work on the approximation capabilities of feedforward neural networks with a single hidden layer of arbitrary width and arbitrary activations. Universal approximation with networks of fixed width but arbitrary depth is studied in \cite{HaninSellke2017a,LuPuWangEtAl2017a,KidgerLyons2020a,ParkYunLeeEtAl2021a}. Efficiency/optimality of approximation is studied in \cite{Barron1994a,Telgarsky2016a,Yarotsky2018a,ElbrachterPerekrestenkoGrohsEtAl2021a,WangQu2022a}. Differing function classes, domains, error rates, network structures, activation functions, etc, are considered.

 A recurrent neural network (RNN) incorporates feedback from the output of some activation functions to the input of others with, say, a unit time delay \cite[e.g. Figures 1 and 2]{Elman1990a}. Thus, RNN's can model dynamical systems with feedback and external inputs; i.e. iterations of a map of an input \textit{and} the state of the system at a prior time, to the output state at the current time \cite{Elman1990a,Haykin2009a}. This article concerns only discrete-time RNNs, e.g. as in \cite{PascanuGulcehreChoEtAl2014a}.

We pause to distinguish two high-level concepts of approximation accuracy in the context of RNNs. The first concerns \textit{map approximation}: in this case, the RNN is frozen in time and the accuracy of the map from the prior time state of the network and the input space, to the output space or current state of the network is appraised (relative to some ground truth). Generally, map approximation is just a reformulation of the approximation theory of feedforward neural networks \cite{Doya1993a}. The second concerns \textit{trajectory approximation}: in this case, the accuracy of the RNN's output trajectory, on some interval, relative to some ground truth trajectory is appraised. Accuracy in this case is a product of both the map approximation error and the accumulation of that error over time due to iterated compositions.

The capability of RNNs to approximate discrete-time state-space models and trajectories on finite time intervals is studied in \cite{Li1992a,Sontag1992a,Doya1993a,JinNikiforukGupta1995a,SchaeferZimmermann2007a,Patan2008a,KimPatronBraatz2011a}. Map approximation on its own is studied in \cite{Doya1993a}; but such results follow directly from the approximation capabilities of feedforward networks. The article \cite{JinNikiforukGupta1995a} studies the approximation of certain trajectories of state-space models with bounded inputs. The idea in \cite{JinNikiforukGupta1995a} relies strictly on the assumption of a finite time interval of interest. In that case, the trajectory is constrained to a compact set and a continuous map may be assumed to be Lipschitz. Any error due to map approximation then grows at most in time (i.e. through repeated map iterations) by a factor determined by the maximum approximation error in a single-step (taken over a compact set) and the iterated Lipschitz constant of the map. By picking the single-step map approximation error small enough, the accumulated error can be bounded below a desired threshold when the horizon is given and finite. The results in \cite{SchaeferZimmermann2007a,Patan2008a,KimPatronBraatz2011a} are mostly the same.

 In this article we study \textbf{universal \textit{time-uniform} trajectory approximation}: i.e. we consider the properties of very general (random) dynamical systems and the RNN characteristics that enable trajectory approximation to any desired accuracy, uniformly in time, over an indefinite or infinite time interval. The trick in this case is to ensure the map approximation errors are uniformly controlled in space and the error incurred at each time step is forgotten fast enough so that the total error does not accumulate unbounded. It is really the imposition of conditions on the dynamical system to be approximated that facilitate the result in this article.

The work in \cite{Matthews1993a,MatthewsMoschytz1994a} considered the problem of time-uniform trajectory approximation with RNNs for deterministic discrete-time dynamical systems on compact domains with bounded inputs. A stability condition is imposed on the system approximated such that the approximation error incurred at any single time is forgotten fast enough so that the sum of errors does not grow unbounded. The condition imposed on the system in \cite{Matthews1993a} is a type of so-called fading memory condition, see \cite{BoydChua1985a}; and it is essentially a type of contractive stability condition. See also \cite{Sandberg1991a} where stronger conditions are imposed. Fading memory-type conditions were later applied to derive time-uniform trajectory approximation results in the setting of so-called echo state networks (i.e. RNNs in which most hidden layer parameters are chosen at random and only the output layer parameters are learnt in training), see \cite{GrigoryevaOrtega2018b,GrigoryevaOrtega2018a,HartHookDawes2020a,GononOrtega2020a,GononOrtega2021a}.

Approximation of continuous-time trajectories (e.g. ordinary differential equations) with continuous time RNNs is studied in \cite{Sontag1992a,FunahashiNakamura1993a,KosmatopoulosPolycarpouChristodoulouEtAl1995a,ChowLi2000a,LiHoChow2005a}. That work is analogous to \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a}; assuming a fixed finite time horizon, trajectories restricted to a compact set, and a map approximation error small enough so that the accumulated error (which grows, typically exponentially) remains below a desired threshold over the finite horizon. Time-uniform approximation of stable ordinary differential equations with bounded inputs using continuous-time RNNs is introduced in \cite{HansonRaginsky2020a}; with trajectories restricted to a compact set. A special case of continuous-time approximation of periodic orbits is considered in \cite{NakamuraNakagawa2009a}.

With regards to time-uniform approximation, we comment further on \cite{Matthews1993a}, the work in echo state networks \cite{GrigoryevaOrtega2018a,GononOrtega2020a}, and the continuous-time setting in \cite{HansonRaginsky2020a}, in a later section; after which the notation, certain technical concepts, and the main result and its proof can be easily referenced.

There are considerable differences in the approach taken in this article, when compared with existing literature. We remark on some characteristics of the formulation, methods, and results in this article worth noting here. We consider discrete-time dynamical systems and trajectories on non-compact spatial domains and over indefinite/infinite time intervals. We consider random dynamical systems with (random) unbounded inputs. We consider approximation in the natural sense of $p$-moments. This theory does not preclude the special case of deterministic state-space models, or trajectories evolving in some compact set. However, it is more natural in some common applications of RNN-based trajectory approximation to consider the formulation herein. The conditions imposed are minimal, testable, and the proof is simple and transparent in its application of the conditions so as to yield some pedagogical value. The formulation and presentation is framed in the language of (random) dynamical systems and state-space models. We provide examples, both empirical and derived, that illustrate the natural satisfaction of the assumptions and motivate the result. As a side project, we also give a different network construction for universal approximation of finite length trajectories that complements \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a}.

We postpone further discussion until later; when more detail can be referenced. Later we discuss the practical relevance (in design/training) and applicability of this work; we comment on the main result and its relationship to the perturbation theory of dynamical systems, and to related work \cite{Matthews1993a,GrigoryevaOrtega2018a,GononOrtega2020a,HansonRaginsky2020a}; we touch on the approximation of Lipschitz maps; and we contrast this work with trajectory approximation on finite time intervals and the relevance of this distinction; and we discuss extensions.

\section{Dynamical Systems}

Let $\mathbb{X}\subseteq\mathbb{R}^{d_x}$ and let $\|\cdot\|:\mathbb{X}\rightarrow[0,\infty)$ be any norm on $\mathbb{X}$. Let $(\mathbb{X}, \mathcal{B}(\mathbb{X}))$ be a standard Borel space. Let $t\in\mathbb{N}:=\{0,1,2,\ldots\}$. Consider the (Borel) measurable endomorphism $f:\mathbb{X}\rightarrow\mathbb{X}$. A dynamical system \cite{CornfeldFominSinai1982a} is just the compositional iteration of $f$, acting on some initial point $x_0\in\mathbb{X}$, that is,
\begin{equation}
	x_t \,=\,  f(x_{t-1}) \,=\,  f^t(x_0)  \label{recursiveSysDef}
\end{equation}
with $f^0=\mathrm{Id}$; and composition often written as multiplication. The solution $\{f^t(x_0) \,|\,t\in\mathbb{N}\}$ traces a sequence of points in $\mathbb{X}$ which we call the \textit{trajectory} of the dynamical system from $x_0\in\mathbb{X}$.

The Polish space of all probability measures on $\mathbb{X}$ is denoted by $\mathcal{P}(\mathbb{X})$. On $(\mathcal{P}(\mathbb{X}), \mathcal{B}(\mathcal{P}(\mathbb{X}))$ we have the push-forward operation $f_\#\mu(A) := \mu(f^{-1}(A))$, $\forall A\in\mathcal{B}(\mathbb{X})$ with $f^0_\#:=\mathrm{Id}$. Treating measures $\mu_t\in\mathcal{P}(\mathbb{X})$ as points in a Polish space, we write, as in (\ref{recursiveSysDef}), the shorthand for the dynamical system,
\begin{equation}
	\mu_t \,=\, f_\#\mu_{t-1} \label{recursiveSysMeas}
\end{equation}
and consider the trajectory $\{f^t_\#\mu_{0}  \,|\,t\in\mathbb{N}\}$ of points in $\mathcal{P}(\mathbb{X})$ for a sequence $t\in\mathbb{N}$.

An \textit{invariant measure} for $f$ is any $\mu\in\mathcal{P}(\mathbb{X})$ with $\mu(f^{-1}(A)) = \mu(A)$ for all $A\in\mathcal{B}(\mathbb{X})$. An invariant measure is a fixed-point trajectory $\mu = f^t_\#\mu$ for all $t\in\mathbb{N}$ in $\mathcal{P}(\mathbb{X})$. Fixing the space $(\mathbb{X},\mathcal{B},\mu)$ first, a dynamical system is called \textit{measure-preserving} if $\mu = f^t_\#\mu$. See \cite{CornfeldFominSinai1982a} for general background.

\subsection{Random Dynamical Systems}

Fix throughout the standard probability space $(\Omega,\mathcal{B}(\Omega),\mathsf{P})$. Consider the $\mathfrak{L}{\mathrm{ip}}(\mathbb{X})$-valued\footnote{We denote by $\mathcal{L}\mathrm{ip}(\mathbb{X})$ the space of Lipschitz endomorphisms on $\mathbb{X}$ with a finite Lipschitz constant.} random process defined by the indexed family $\{F_t \,|\, t\in\mathbb{N}\}$ of measurable maps $F_t : \Omega\rightarrow\mathfrak{L}{\mathrm{ip}}(\mathbb{X})$. We write $F_t(\omega)(\cdot):\mathbb{X}\rightarrow\mathbb{X}$ for a realised $\omega\in\Omega$ sample function. This process is well defined, see e.g. \cite{DiaconisFreedman1999a}.

A random dynamical system on $\mathbb{X}$ is defined by the compositional iteration of random (Lipschitz) endomorphisms $F_t$ acting on some initial $x_0\in\mathbb{X}$,
\begin{equation}
	X^{x_0}_t \,=\,  F_{t} \cdots  F_2 \cdot F_1(x_0) \,=\,  F_{t}(X^{x_0}_{t-1}) \label{recursiveRandSysDef}
\end{equation}
The indexed family $\{X_t^{x_0} \,|\,t\in\mathbb{N},x_0\in\mathbb{X} \}$ of random variables $X^{x_0}_t:\Omega\rightarrow\mathbb{X}$ is an induced random process\footnote{We write $X^{\mu_0}_t$ if $X_0:\Omega\rightarrow\mathbb{X}$ is random with distribution $\mu_0\in\mathcal{P}(\mathbb{X})$. In general we write $X'\sim\mu$ if $X'$ is random with law $\mu\in\mathcal{P}(\mathbb{X})$. We write $X^{X'}_t$ if $X_0=X'$ almost surely. We often drop reference to the initial condition.} on $\mathbb{X}$.  The realised solution $\{X_t^{x_0}(\omega) \,|\,t\in\mathbb{N}\}$ is the \textit{trajectory} of the random dynamical system from the state $x_0\in\mathbb{X}$ with $\omega\in\Omega$.

Let $\mathbb{U}\subseteq\mathbb{R}^{d_u}$. Let $(\mathbb{U}, \mathcal{B}(\mathbb{U}))$ be a standard Borel space. The indexed family $\{U_t \,|\, t\in\mathbb{N}\}$ of random variables $U_t:\Omega\rightarrow\mathbb{U}$ is a random process. We consider the maps in (\ref{recursiveRandSysDef}) of the form,
\begin{equation}
	F_{t}(\omega)(\cdot) \,:=\, f(\cdot,U_{t}(\omega))  \label{recursiveRandSysInputDef}
\end{equation}
where $f:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ is a deterministic, time-invariant, measurable map, Lipschitz in $\mathbb{X}$.

See \cite{Ohno1983a,ArnoldCrauel1992a,DiaconisFreedman1999a,Arnold2003a} for further background on these systems. Note that an independent and identically distributed sequence of maps does not induce an independent process in $X_t$, but it is Markov \cite{Kifer1986a}; and, a stationary Markov sequence of maps does not induce a Markov process in $X_t$. A stationary process of certain contractive maps is, with the right initialisation, a stationary process in $X_t$ \cite{Elton1990a}.

\begin{lemma}\cite{Kifer1986a}\label{markoviidlemma}
	The process $\{X_t^{x_0}\,|\,t\in\mathbb{N}\}$ in (\ref{recursiveRandSysDef}) is Markov for all $x_0\in\mathbb{X}$ if and only if the sequence of random maps $\{F_t \,|\, t\in\mathbb{N}\}$ is independently distributed. A Markov process $\{X_t^{x_0} \,|\,t\in\mathbb{N}\}$ in (\ref{recursiveRandSysDef}) is homogenous for all $x_0\in\mathbb{X}$ if and only if the sequence $\{F_t \,|\, t\in\mathbb{N}\}$ is identically distributed.
\end{lemma}

Consider an invertible measure-preserving endomorphism $\theta:\Omega\rightarrow\Omega$ on $(\Omega,\mathcal{B}(\Omega),\mathsf{P})$. If $F_t(\omega) = F(\theta^{t}(\omega))$ for a random $F : \Omega\rightarrow\mathfrak{L}{\mathrm{ip}}(\mathbb{X})$, then $\{F_t \,|\, t\in\mathbb{N}\}$ is stationary \cite[Appendix A]{Arnold2003a}. An extended\footnote{It is possible to extend a sequence of stationary random maps $\{F_t \,|\, t\in\mathbb{N}\}$ over all the integers $\mathbb{Z}$ such that the doubly infinite process $\{F_t \,|\, t\in\mathbb{Z}\}$ is stationary and the law of any combination of maps with non-negative time indices in the extended sequence is the same as in the original sequence; e.g. see \cite[Lemma 1]{Elton1990a}, Doob \cite[page 456]{Doob1953a}, or \cite[Appendix A]{Arnold2003a}.} stationary sequence $\{F_t \,|\, t\in\mathbb{Z}\}$ is identically distributed (but not vice versa).

The Lipschitz constant of $F\in\mathfrak{L}{\mathrm{ip}}(\mathbb{X})$ is the smallest $\mathfrak{L}{(F)}:\Omega\rightarrow[0,\infty)$ such that,
\begin{equation}
	\|F(\omega)(x) - F(\omega)(x_0 )\| \,\leq\, \mathfrak{L}{(F)}(\omega)\,\|x - x_0\|
\end{equation}
A particular Lipschitz map is called \textit{contractive} if its Lipschitz constant is $<1$. Loosely speaking, if the sequence $\{F_t \,|\, t\in\mathbb{Z}\}$ is stationary and over time the maps are contracting in some average sense, then the induced process $\{X_t\,|\,t\in\mathbb{N}\}$ converges in distribution to a stationary process. The induced process also exhibits some contractive stability and initial states are forgotten. For example, if $\{F_t \,|\, t\in\mathbb{Z}\}$ is independent and identically distributed, then the condition $\mathsf{E}[\log\mathfrak{L}{(F)}]<0$ formalises a notion of \textit{contracting in the average}, and leads to the stated conclusions; see \cite{DiaconisFreedman1999a}. We will not generally impose an independence assumption on the maps, i.e. we do not assume $\{X_t\,|\,t\in\mathbb{N}\}$ is Markov. We also assume a weaker notion, in some directions at least, of contracting in the average. The following result is an extension of a classical result of Elton \cite{Elton1990a}, see also \cite{DebalyTruquet2021a}.

\begin{lemma}\label{eltonlemma}
Assume $\{F_t(\omega) = F(\theta^{t}(\omega)) \,|\, t\in\mathbb{Z}\}$ is stationary and $\mathsf{E}[\|F(x)-x\|^p]<\infty$ for one $x\in\mathbb{X}$ and some $p\geq1$. Assume
there is a finite $C>0$, and $\lambda\in(0,1)$, all independent of $s,t\in\mathbb{N}$, such that,
\begin{align}
	\mathsf{E} \big[ \,\| F_{t}  \cdots  F_{s+1}(X^{x}_s) - F_{t}  \cdots  F_{s+1}({X}^{x_0}_s) \|^p \,\big]
	  ~&\leq~ C\,\lambda^{(t-s)}\, \mathsf{E} \big[\, \|{X}^{x}_{s} - {X}^{x_0}_{s} \|^p \,\big] \label{stabilitycond}
\end{align}
exists for all $x,x_0\in\mathbb{X}$ and all $t> s$. Then there is a random variable, $X_\infty:\Omega\rightarrow\mathbb{X}$, such that,
 \begin{equation}
		\lefteqn{\overbrace{\phantom{\lim_{t\rightarrow\infty}~X_t^{x_0} \,=\, \lim_{t\rightarrow\infty} ~F_{t} \cdots  F_2 \cdot F_1(x_0) ~\stackrel{\text{law}}{=}~ X_\infty}}^{forward~convergence~in~distribution}}\lim_{t\rightarrow\infty}~X_t^{x_0} \,=\, \lim_{t\rightarrow\infty} ~F_{t} \cdots  F_2 \cdot F_1(x_0) ~\stackrel{\text{law}}{=}~ \underbrace{X_\infty~=~ \lim_{t\rightarrow\infty} ~F_{0}\cdot F_{-1}  \cdots  F_{-t}(x_0)}_{backward~almost~sure~convergence} \label{convergencelemmaeqn}
\end{equation}
and $X_\infty$ is independent of $x_0$, i.e, $\mathsf{P}(X_\infty(x_0)\neq X_\infty(x))=0$. The process $\{X_t^{\mu_0}\,|\,t\in\mathbb{N}\}$ with $\mu_0:=\mathrm{Law}(X_{\infty})$ is stationary in $\mathbb{X}$. If $\{F_t \,|\, t\in\mathbb{N}\}$ is also ergodic, then $\{X_t \,|\, t\in\mathbb{N}\}$ is ergodic.
\end{lemma}

The condition of interest is (\ref{stabilitycond}) which asks for the composition of maps to \textit{eventually be contracting on average}; e.g. see the terminology in \cite{Steinsaltz1999a,WuShao2004a}. If $\{F_t \,|\, t\in\mathbb{N}\}$ is also independent, then $\{X_t^{x_0}\,|\,t\in\mathbb{N}\}$ is Markov and the backward limit is given by $X_\infty = \lim_{t\rightarrow\infty} ~F_{1}\cdot F_{2}  \cdots  F_{t}(x_0)$, see \cite{DiaconisFreedman1999a}. Variations on the type of average contraction, e.g. on \eqref{stabilitycond}, that lead (mostly) to the same conclusions are studied in the literature, particularly when $\{X_t\,|\,t\in\mathbb{N}\}$ is Markov, or when $F_t$ is affine or takes values in a finite subset of $\mathfrak{L}{\mathrm{ip}}(\mathbb{X})$. See \cite{BarnsleyElton1988a,Elton1990a, ArnoldCrauel1992a,BurtonRoesler1995a,Steinsaltz1999a,WuShao2004a,Stenflo2012a}.

A sequence of random maps $\{F_t \,|\, t\in\mathbb{N}\}$, not necessarily stationary, that satisfies \eqref{stabilitycond} is called \textit{exponentially $p$-contractive}.

\subsection{Example: Complex Trajectories from Simple Dynamics} \label{barnsleysfernexample1}

Let $\Omega=\{1,\ldots,K\}^\mathbb{N}$ with finite $K\in\mathbb{N}$, let $\mathsf{p}=(p_1,\ldots,p_K)$, $\sum_ip_i=1$, and let $\mathsf{P}=\mathsf{p}^\mathbb{N}$. Each $\omega\in\Omega$ has representation $\omega=(\omega_0,\omega_1,\ldots)$. Let $\theta:\Omega\rightarrow\Omega$ be the left shift $\theta^t(\omega)=(\omega_t,\omega_{t+1}\ldots)$, defining a measure preserving $\theta^t_\#\mathsf{P}=\mathsf{P}$ dynamical system on $(\Omega,\mathcal{B}(\Omega),\mathsf{P})$.
Let $\{G_i:\mathbb{X}\rightarrow\mathbb{X}\,|\,i\in\{1,\ldots,K\}\}$ be a collection of deterministic Lipschitz maps. Define a random map by,
\begin{align}
	F: \Omega\rightarrow \{G_1,\ldots,G_K\} \subset \mathfrak{L}{\mathrm{ip}}(\mathbb{X}) \qquad\mathrm{with}\qquad
	\theta^t(\omega)\mapsto F(\theta^t(\omega)) = G_{\omega_t}
\end{align}
The sequence $\{F_t(\omega) = F(\theta^{t}(\omega)) \,|\, t\in\mathbb{N}\}$ is an independent and identically distributed sequence. For any $x\in\mathbb{X}$, an induced process is generated by independently picking $G_i$ with probability $p_i$ and moving to $G_i(x)$. The induced process $X^{x_0}_t = F_{t} \cdots  F_2 \cdot F_1(x_0)$, see (\ref{recursiveRandSysDef}), is a Markov chain.

Remarkably complicated and interesting induced trajectory behaviour may arise from this seemingly simple system, even in low dimension with $K$ small. For example, let $\mathbb{X}=\mathbb{R}^2$, $K=4$ and,
\begin{align}
	G_1(x) \,=\, \begin{bmatrix}\ 0.00&\ 0.00\ \\0.00&\ 0.16\end{bmatrix}x+{\begin{bmatrix}\ 0.00\\0.00\end{bmatrix}},\quad G_2(x) \,=\, {\begin{bmatrix}\ 0.85&\ 0.04\ \\-0.04&\ 0.85\end{bmatrix}}x+{\begin{bmatrix}\ 0.00\\1.60\end{bmatrix}}, \\
	G_3(x) \,=\,  {\begin{bmatrix}\ 0.20&\ -0.26\ \\0.23&\ 0.22\end{bmatrix}}x+{\begin{bmatrix}\ 0.00\\1.60\end{bmatrix}}, \quad G_4(x) \,=\, {\begin{bmatrix}\ -0.15&\ 0.28\ \\0.26&\ 0.24\end{bmatrix}}x+{\begin{bmatrix}\ 0.00\\0.44\end{bmatrix}}
\end{align}
with $\mathsf{p}=(0.01, 0.85, 0.07, 0.07)$. Lemma \ref{eltonlemma} trivially applies here. Set $x_0=0$ and iterate the induced random process over, say, $T=1000$, $T=10000$ and $T=100000$ time-steps. The resulting empirical trajectory in phase space is plotted in Figure \ref{fig:barnsleysfern}, and is known as Barnsley's fern \cite{BarnsleyElton1988a}.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.75\textwidth]{barnsleysfern.pdf}
    \caption{The trajectory of Barnsley's fern process with $T=1000$, $T=10000$ and $T=100000$.}
    \label{fig:barnsleysfern}
\end{figure}

We do not attempt to survey the rich and interesting nature of this type of random dynamical system or its ergodic and invariant measure properties; see e.g. \cite{Hutchinson1981a,BarnsleyElton1988a}. We simply note that the forward induced process generating Barnsley's fern does not converge to any point (obviously) but instead wanders ergodically through the state space in accordance with its unique invariant measure (which happens to place most mass in the shape of the depicted fern, e.g. as in the right-most image in Figure \ref{fig:barnsleysfern}).

\subsection{Example: Contraction on Average with Non-Contractive Maps}

Assume the independent setup in Section \ref{barnsleysfernexample1} and consider two non-contractive maps,
\begin{align}
	G_1(x) \,=\, \begin{bmatrix}\ 10/9 &\ 0\\ \ 0&\ 1/2\end{bmatrix}x +\mathbf{b}_1 \qquad G_2(x) \,=\, {\begin{bmatrix}\ 1/2&\ 0\ \\0&\ 10/9\end{bmatrix}}x +\mathbf{b}_2
\end{align}
with $\mathsf{p}=(0.5, 0.5)$ and any $\mathbf{b}_1$, $\mathbf{b}_2\in\mathbb{R}^2$. Then for any two initial points $x,x_0\in\mathbb{X}$ consider,
\begin{align}
	\mathsf{E} \big[ \, \|X_1^{x} - X_1^{x_0}\|  \,\big] \,\leq\,   \left(\frac{1}{2}\|\mathbf{A}_1\|\ + \frac{1}{2}\|\mathbf{A}_2\|\right)\|x-x_0\| \,=\, \frac{10}{9}\|x-x_0\|
\end{align}
with $G_i(x)=\mathbf{A}_ix+\mathbf{b}_i$. The map is non-contracting on average in a single iteration. However,
\begin{align}
	\mathsf{E} \big[ \, \|X_2^{x} - X_2^{x_0}\|  \,\big] \,\leq\,   \left(\frac{1}{4}\|\mathbf{A}_1\mathbf{A}_1\|\ +\frac{1}{2}\|\mathbf{A}_1\mathbf{A}_2\|+ \frac{1}{4}\|\mathbf{A}_2\mathbf{A}_2\|\right)\|x-x_0\| \,<\, \frac{9}{10}\|x-x_0\|
\end{align}
and the maps are contracting, on average, over two iterations. Necessary and sufficient conditions, replacing (\ref{stabilitycond}), for the conclusions of Lemma \ref{eltonlemma} to hold in the setting of random affine maps are given in \cite{BougerolPicard1992a}; and take advantage of a logarithm inside the expectation. Here we just provide a basic example for (\ref{stabilitycond}) to hold with each map non-contracting; much more complicated examples are possible.

This example allows one to contrast the property of exponential $p$-contractivity, an average-type contraction property, with the deterministic fading memory-type property considered in \cite{BoydChua1985a,Matthews1993a,GrigoryevaOrtega2018a}.

\section{Recurrent Neural Network Architectures}

Let $\sigma(\cdot):=\max\{\cdot,0\}:\mathbb{R}\rightarrow\mathbb{H}\subseteq\mathbb{R}$; which acts component wise on a vector $v\in\mathbb{R}^d$ so $\sigma(v)\in\mathbb{H}^d$.

A general deep recurrent neural network (RNN), with $L\in\mathbb{N}$ layers, is given in the form,
\begin{align}
	\widehat{X}_t \,=\, {h}_{L,t} &\,=\, \tau_L({h}_{L-1,t}) \label{RNNform1} \\
	{h}_{l,t} &\,=\, \sigma\left(\tau_l({h}_{l-1,t}) + \varphi_l({h}_{t-1}) \right),\qquad l\in\{1,\ldots,L-1\} \label{RNNform2} \\
	{h}_{0,t} &\,:=\, U_{t}  \label{RNNform3}
\end{align}
where ${h}_{l,t}\in\mathbb{H}^{d_{h_l}}$ with $\mathbb{H}^{d_{h_0}}:=\mathbb{U}$ and $\mathbb{H}^{d_{h_L}}:=\mathbb{X}$. The feedback vector is ${h}_{t}=({h}_{1,t},\ldots,{h}_{L,t})\in\mathbb{H}^{d_h}$. The maps
\begin{equation}
	\tau_l:\mathbb{H}^{d_{h_{l-1}}}\rightarrow\mathbb{R}^{d_{h_{l}}}  \qquad\mathrm{and}\qquad \varphi_l:\mathbb{H}^{d_h}\rightarrow\mathbb{R}^{d_{h_{l}}}
\end{equation}
are finite affine maps.

We write a generic deep RNN of the form (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}) as,
\begin{equation}
	\widehat{X}^{h_0}_t \,=\, \widehat{f}({h}_{t-1}, U_{t}) \,=:\, \widehat{F}_{t}({h}_{t-1}) \label{genericformRNN}
\end{equation}
Specifying the parameters of the affine maps defines the topology of the network $\widehat{f}$. Specifying also the initial condition $h_0\in\mathbb{H}^{d_h}$ defines the resulting RNN output for a given input sequence.

The form (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}) is very general, covering at least all those models in \cite{PascanuGulcehreChoEtAl2014a}. We will not construct any networks utilising this general form. Common specialisations of this structure (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}) involve feedback solely from the last activation layer to the first layer, and feedback from the output of each activation layer to itself, see \cite{PascanuGulcehreChoEtAl2014a}.

Feedback from the last layer, or equivalently the last hidden layer, in (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}) amounts to a recursion on $\widehat{X}_t$. Consider the special form,
\begin{align}
	\widehat{X}_t \,=\, \tau_L\cdot\sigma\cdot\tau_{l-1}\cdots\sigma\cdot\tau_2\cdot\sigma(\tau_1(U_t)  + \varphi(\widehat{X}_{t-1}) ) \label{RNNspecialform}
\end{align}
where $\varphi:\mathbb{X}\rightarrow\mathbb{R}^{d_{h_1}}$ is a finite affine feedback map. This form (\ref{RNNspecialform}) of RNN is written as,
\begin{equation}
	\widehat{X}_t \,=\, \widehat{f}(\widehat{X}_{t-1}, U_{t}) \,=:\, \widehat{F}_{t}(\widehat{X}_{t-1}) \label{genericspecialformRNN}
\end{equation}

\subsection{Map Approximation versus Trajectory Approximation}

Map approximation seeks for any $\epsilon>0$ a network $\widehat{f}({h}, \cdot)$ and a network state ${h}\in\mathbb{H}^{d_h}$ such that,
\begin{equation}
	\mathsf{E}\big[ \,\| \widehat{f}({h},\cdot) -  {f}(x,\cdot) \|^p\, \big]^{1/p} \,\leq\, \epsilon
\end{equation}
for all $x$ and all inputs in $\mathbb{X}\times\mathbb{U}$. The network initialisation is explicitly a part of the approximation. Map approximation theory follows from classical universal approximation results for feedforward networks, e.g. by considering the RNN frozen at a single time. In the proof of our main theorem we call on a map approximation result: we state an approximation theorem for feedforward neural networks in Appendix \ref{UAFFappendix}. In general, repeated composition of an approximate map may lead to an accumulation of error.

The problem of \textit{trajectory approximation} seeks for any $\epsilon>0$ a network $\widehat{f}$ and a network initialisation ${h}_0\in\mathbb{H}^{d_h}$ such that,
\begin{equation}
	\mathsf{E}\big[ \,\big\| \widehat{F}_{t}\cdots\widehat{F}_2\cdot\widehat{F}_1({h}_0) \,-\, {F}_{t}\cdots{F}_2\cdot {F}_1(x_0) \big\|^p\, \big]^{1/p} \,\leq\, \epsilon
\end{equation}
for all times in a given interval $t\in\{0,\ldots,T\}$, $T\in\mathbb{N}$ and for all $x_0$ and all inputs in $\mathbb{X}\times\mathbb{U}^T$. Trajectory approximation over known fixed-length intervals is studied in \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a}. These results are generally unsuitable for trajectories evolving on non-compact domains and over indefinite/infinite horizons. In Appendix \ref{appendixFiniteTime} we prove another finite-horizon universal trajectory approximation result, based on a significantly different network construction and proof than considered in previous works, e.g. \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a}. This construction complements prior work and may be of general interest. The result in Appendix \ref{appendixFiniteTime} also accommodates more general dynamical system models than previously considered.

Finally, we call a trajectory approximation error bound that holds $\underline{\underline{\mathrm{for~all}~t\in\mathbb{N}}}$ a \textit{time-uniform} trajectory approximation error bound.

\section{Universal Time-Uniform Trajectory Approximation}

The following is the main result of this article.

\begin{theorem}\label{theoremuniversaltrajapprox}
	Consider the map $f:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ of (\ref{recursiveRandSysInputDef}) and a RNN $\widehat{f}$ of the form (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}), see also (\ref{genericformRNN}). Let $X_0:\Omega\rightarrow\mathbb{X}$ be random, independent, and have finite $p\geq1$ moments. Assume $\{F_t(\omega) = F(\theta^t(\omega))\,|\, t\in\mathbb{N}\}$ is stationary,  and exponentially $p$-contractive, and $\mathsf{E}\left[\|F(x)-x\|^p\right]<\infty$. Then for any $\epsilon>0$ there is a finite $\widehat{f}$ and a random initial $H_0:\Omega\rightarrow\mathbb{H}^{d_h}$ such that,
\begin{equation}
		\mathsf{E}\big[ \,\big\| \widehat{F}_{t}\cdots\widehat{F}_2\cdot\widehat{F}_1(H_0) \,-\, {F}_{t}\cdots{F}_2\cdot {F}_1(X_0) \big\|^p\, \big]^{{1}/{p}}  ~\leq~ \epsilon
\end{equation}
for all $t\in\mathbb{N}$ and the sequence $\{\smash{\widehat{F}_t}(\cdot):=\smash{\widehat{f}(\cdot,U_t)} \,|\, t\in\mathbb{N}\}$ is exponentially $p$-contractive.
\end{theorem}

The conditions of interest are stationarity and exponential contractivity of $\{F_t(\cdot)=f(\cdot,U_t) \,|\, t\in\mathbb{N}\}$. The induced process $\{X_t\,|\, t\in\mathbb{N}\}$ is not necessarily Markov, but it is asymptotically stationary under the hypotheses of the theorem, via Lemma \ref{eltonlemma}.

\begin{proof}
Consider a RNN of the form (\ref{RNNspecialform}); also written as (\ref{genericspecialformRNN}). This is a (very) special case of the more general (\ref{RNNform1}), (\ref{RNNform2}), (\ref{RNNform3}). Consider the error,
\begin{equation}
		 \widehat{X}_t^{h_{0}} \,-\, X_t^{x_0}  \,=\, \widehat{F}_{t}\cdots\widehat{F}_2\cdot\widehat{F}_1({h_0}) \,-\, {F}_{t}\cdots{F}_2\cdot {F}_1(x_0)
\end{equation}
for any ${x}_0\in\mathbb{X}$ and with $h_0=h_{L,0}\in\mathbb{X}$ in the special case (\ref{RNNspecialform}). Let $h_{L,0}=x_0$. Then we can expand,
\begin{align}
		 \widehat{X}_t^{{x_0}} \,-\, X_t^{x_0}  \,&=\, \sum_{s=1}^{t} {F}_{t}\cdots{F}_{s+1} \cdot \widehat{F}_{s}\cdot\big(\widehat{F}_{s-1}\cdots\left({x}_0\right)\big) \,-\, {F}_{t}\cdots{F}_{s+1} \cdot {F}_{s}\cdot\big(\widehat{F}_{s-1}\cdots\left({x}_0\right)\big) \nonumber \\
		 &=\, \sum_{s=1}^{t} \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot \widehat{F}_{s}\cdot\big({F}_{s-1}\cdots\left({x}_0\right)\big) \,-\, \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot {F}_{s}\cdot\big({F}_{s-1}\cdots\left({x}_0\right)\big)
\end{align}
This is a telescopic expansion (easy to verify, e.g. with $t\in\{3,4\}$). Writing $X_{s-1}^{x_0}={F}_{s-1}\cdots({x}_0)$, we can take norms $\|\cdot\|_p:=\mathsf{E}[\|\cdot\|^p]^{1/p}$,
\begin{align}
	\big\| \widehat{X}_t^{{x}_0} - X_t^{x_0} \big\|_p &=\, \big\|  \sum_{s=1}^{t} \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) - \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p \nonumber\\
		&\leq\, \sum_{s=1}^{t}   \,\big\| \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) - \widehat{F}_{t}\cdots \widehat{F}_{s+1} \cdot {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p \label{normederrordecomposition}
\end{align}
Assume that the sequence of random maps $\{\widehat{F}_t \,|\, t\in\mathbb{N}\}$ is exponentially $p$-contractive. We will verify this assumption later. With this assumption we can write the last line as,
\begin{align}
		\big\| \widehat{X}_t^{{x_0}} - X_t^{x_0} \big\|_p
		\,&\leq\,  \big\| \widehat{F}_{t}\big(X_{t-1}^{x_0}\big) - {F}_{t}\big(X_{t-1}^{x_0} \big) \big\|_p   \,+\, C\sum_{s=1}^{t-1} \lambda^{(t-s-1)} \big\| \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) - {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p \label{sumsinglestep}
\end{align}
This is a sum of single-step approximation errors (arising as a result of the RNN map approximation) weighted by the stability index due to the exponential contractiveness of $\{\smash{\widehat{F}_t} \,|\, t\in\mathbb{N}\}$.

Let $X'\sim\mu'\in\mathcal{P}(\mathbb{X})$ be some random variable. We will consider $\| \widehat{F}_{s}(X_{s-1}^{x_0}) - {F}_{s}(X_{s-1}^{x_0} ) \|_p$ for any $s\in\{1,\ldots,t\}$. Applying the triangle inequality twice we get,
\begin{align}
		\big\| \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) \,-\, {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p \,&\leq\, \big\| \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) \,-\, \widehat{F}_{s}\big(X_{s-1}^{X'} \big) \big\|_p  + \big\| \widehat{F}_{s}\big(X_{s-1}^{X'}\big) \,-\, {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p \nonumber \\
		\,&\leq\, \big\| \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) \,-\, \widehat{F}_{s}\big(X_{s-1}^{X'} \big) \big\|_p + \big\| \widehat{F}_{s}\big(X_{s-1}^{X'}\big) \,-\, {F}_{s}\big(X_{s-1}^{X'} \big) \big\|_p \nonumber \\
		& \qquad + \big\| {F}_{s}\big(X_{s-1}^{X'}\big) \,-\, {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p   \label{singlestepdecomposition}
\end{align}
Since $\{{F}_t \,|\, t\in\mathbb{N}\}$ and $\{\widehat{F}_t \,|\, t\in\mathbb{N}\}$ are exponentially $p$-contractive (the latter to be verified) we have,
\begin{align}
	\big\| \widehat{F}_{s}\big(X_{s-1}^{x_0}\big) - \widehat{F}_{s}\big(X_{s-1}^{X'} \big) \big\|_p \,+\,  \big\| {F}_{s}\big(X_{s-1}^{X'}\big) -  {F}_{s}\big(X_{s-1}^{x_0} \big) \big\|_p ~ \leq~ C^\dagger\, \lambda^{\dagger^{s}}\,  \big\| x_0 - X' \big\|_p
\end{align}
for all $s\in\{1,\ldots,t\}$ and some finite $C^\dagger>0$, $\lambda^\dagger\in(0,1)$. The term $\| x_0 - X' \|_p$ is finite for all $x_0\in\mathbb{X}$, including those random initialisations with finite $p$-th moments.

Now extend the sequence $\{F_t \,|\, t\in\mathbb{N}\}$ to $\{F_t \,|\, t\in\mathbb{Z}\}$. Let
\begin{equation}
	X_\infty~=~ \lim_{t\rightarrow\infty} ~F_{0}\cdot F_{-1}  \cdots  F_{-t}(x)
\end{equation}
almost surely from Lemma \ref{eltonlemma} such that $\{X_t^{\mu}\,|\,t\in\mathbb{N}\}$ with $\mu:=\mathrm{Law}(X_{\infty})\in\mathcal{P}(\mathbb{X})$ is stationary in $\mathbb{X}$.

Setting $\mu'=\mu$, note that $\smash{X_{s}^{X'}}\sim\mu$, $\forall s\in\mathbb{N}$ by stationarity. We then have,
\begin{align}
		\big\| \widehat{X}_t^{{x_0}} - X_t^{x_0} \big\|_p
		 \,\leq\, \left(1+\frac{C}{1-\lambda}\right) \left( C^\dagger\, \lambda^{\dagger^{t}}\,  \big\| x_0 - X \big\|_p \,+\, \big\| \widehat{f}\big(X,U\big) - {f}\big(X,U \big) \big\|_p \right)
\end{align}
where $X\sim\mu$, and $U:\Omega\rightarrow\mathbb{U}$ is distributed by the invariant law of the stationary $\{U_t(\omega)\,|\, t\in\mathbb{N}\}$.

\textbf{Map Approximation}: If we drop the recursion on $t\in\mathbb{N}$ in (\ref{genericspecialformRNN}), then we may view (\ref{RNNspecialform}) as a feedforward neural network, cf. Appendix \ref{UAFFappendix}, with input $(\widehat{X}_{t-1}, U_{t})\in\mathbb{X}\times\mathbb{U}$. Note the sum of the two affine maps $\tau_1$ and $\varphi$ in (\ref{RNNspecialform}) is just another affine map on $(\widehat{X}_{t-1}, U_{t})\in\mathbb{X}\times\mathbb{U}$. This interpretation is all that is needed for the desired map approximation. In this notation $(\widehat{X}_{t-1}, U_{t}) = (X,U)$. Applying Lemma \ref{lemmaUALp} in Appendix \ref{UAFFappendix} it follows that for any $\overline{\epsilon}>0$ there exists a finite neural network $\widehat{f}$ such that,
\begin{equation}
	 \left\| \,\widehat{f}(X,U) - f(X,U)\,  \right\|_p \,\leq\, \overline{\epsilon} \label{mapapproxtheorem1}
\end{equation}
with the added property that $\mathfrak{L}(\widehat{F})(\omega) \leq \mathfrak{L}({F})(\omega)$, $\forall\omega\in\Omega$, where $\widehat{F}(\cdot):=\widehat{f}(\cdot,U)$, $F(\cdot):=f(\cdot,U)$. This completes a map approximation result.

Because $\overline{\epsilon}$ can be fixed arbitrary, we may solve, say, $\epsilon \geq (\overline{\epsilon} + C^\dagger\, \| x_0 - X'_0 \|_p)\,(1+\frac{C}{1-\lambda})$ for $\overline{\epsilon}>0$ and we obtain the desired trajectory approximation result for any initialisation $x_0\in\mathbb{X}$, including random initialisations with finite $p$-th moments. We note the accuracy of map approximation needed to achieve the desired trajectory approximation may depend on the initialisation when it is not at stationarity; however, the error from any non-stationary starting point is forgotten exponentially fast $C^\dagger \lambda^{\dagger^{t}} \smash{ \big\| x_0 - X' \big\|_p}$.

It remains to establish the sequence of maps $\{\widehat{F}_t \,|\, t\in\mathbb{N}\}$ is exponentially $p$-contractive. However, this follows directly from the second part of Lemma \ref{lemmaUALp} in Appendix \ref{UAFFappendix} as $\mathfrak{L}(\widehat{F})(\omega) \leq \mathfrak{L}({F})(\omega)$, $\forall\omega\in\Omega$ implies $\{\smash{\widehat{F}_t} \,|\, t\in\mathbb{N}\}$ is exponentially $p$-contractive whenever $\{{F}_t \,|\, t\in\mathbb{N}\}$ is exponentially $p$-contractive.

This completes the proof of Theorem \ref{theoremuniversaltrajapprox}.
\end{proof}

\subsection{Example: Learning to Generate Fractals from Short Samples} \label{sec:fernPractical}

We return to the example and setup in Section \ref{barnsleysfernexample1}. Let $\mathbb{U}=\{1,\ldots,K\}$. A random variable $U:\Omega\rightarrow\mathbb{U}$ is defined by $U:=\omega_0$ where $\omega_0$ is the first coordinate of $\omega\in\Omega$. The probability is $U_{\#}\mathsf{P}=\mathsf{p}\in\mathcal{P}(\mathbb{U})$. Thus, $\mathsf{P}(\{\omega:U(\omega)=i\}=\mathsf{p}(i)=p_i$. The process $\{U_t(\omega) = U(\theta^t(\omega))\,|\, t\in\mathbb{N}\}$ is independent and identically distributed and acts as a switching input  signal for the switched dynamical system. The induced process from the setup in Section \ref{barnsleysfernexample1} can now be written as,
\begin{equation}
	X^{x_0}_t(\omega) \,=\, f(X^{x_0},U_t(\omega)) \,=\, \left\{\begin{array}{lcl} G_1(X^{x_0}_{t-1}) & & \mathrm{if}~U_t(\omega)=1 \\
															 & \vdots&  \\
															G_K(X^{x_0}_{t-1}) & & \mathrm{if}~U_t(\omega)=K \end{array}\right. \label{ifsU}
\end{equation}
in line with (\ref{recursiveRandSysInputDef}) where $f:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ is non-random, time-invariant, measurable, and Lipschitz in $\mathbb{X}$.

We now consider a simplified version of Barnsley's fern. Let $\mathbb{X}=\mathbb{R}^2$, $x_0=0$, and $K=2$ and consider the affine transformations,
\begin{align}
	G_1(x) \,=\, \begin{bmatrix} 0.40& -0.3733\\ 0.060& 0.60\end{bmatrix}x+{\begin{bmatrix} 0.3533\\0.00\end{bmatrix}},\quad G_2(x) \,=\, \begin{bmatrix} -0.80& -0.1867\\ 0.1371& 0.80\end{bmatrix}x+{\begin{bmatrix} 1.10\\0.10\end{bmatrix}} \label{simpleFern}
\end{align}
with $\mathsf{p}=(0.2993, 0.7007)$. The induced trajectory of (\ref{ifsU}), (\ref{simpleFern}) plots a simplified fern \cite{DiaconisFreedman1999a}.

Consider $N\in\mathbb{N}$ independent trajectories $\{\,(X^{x_0}_t,U_t)_n \,|\, t\in\{1,\ldots,T \},\,n\in\{1,\ldots,N\}\}$. We draw one set of samples for training a RNN; i.e. learning the network parameters (e.g. via back propagation with a mean square error cost \cite{PascanuMikolovBengio2013a}). The initial state is not part of the data in this example. We draw another set of samples for testing the RNN and computing mean error performance. We set $T_\mathrm{train}=50$, $N_\mathrm{train}=1000$, and $T_\mathrm{test}=10000$, $N_\mathrm{test}=2000$. We consider applications of the approximation over significantly longer time intervals than used in training. We train a simple network like (\ref{RNNspecialform}) with a single hidden layer of $6$ scalar activation functions $\sigma(\cdot):=\max\{0,\cdot\}$. See Figure \ref{fig:simpleFern}.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.99\textwidth, height=0.145\textheight]{networkferncase.pdf}
    \caption{An example training, ground truth test and approximated test trajectory is shown; along with the root mean square error on $2000$ test examples, illustrating the time-uniform error control.}
    \label{fig:simpleFern}
\end{figure}

The approximation error introduced at each iteration does not accumulate, nor lead to increasing phase space distortion. This example just serves to illustrate a simple application of Theorem \ref{theoremuniversaltrajapprox}; i.e. we do not consider the training data or ergodic properties of this switched affine system here, nor seek state-of-the-art identification methods. We show later that even in very simple systems a small approximation error can grow and accumulate unbounded when the hypotheses of Theorem \ref{theoremuniversaltrajapprox} are violated.

\subsection{Generalisations to Asymptotically Stationary Sequences of Maps}

Firstly, we emphasise that Theorem \ref{theoremuniversaltrajapprox} is not restricted to Markov processes on $\mathbb{X}$. Asking just for stationarity of the random maps is a more general condition. We will show later a simple application of this result in optimal filtering wherein this additional generality is needed. We can say more.

For the discussion here, lets say a process $\{U_t\,|\,t\in\mathbb{N}\}$ is \textit{asymptotically stationary} if the marginal law of $U_t$ converges toward an invariant measure, and if when $U_0$ is distributed according to this invariant measure, then $U_t$ is stationary. In at least some such cases the conclusions of Theorem \ref{theoremuniversaltrajapprox} remain true. We detail one specific situation here. From (\ref{recursiveRandSysInputDef}), we consider the special case
\begin{equation}
	F_{t}(x) \,=\, f(x,U_{t}) \,:=\, \mathbf{A}\,x \,+\, \mathbf{B}\,U_t  \label{recursiveAffineRandSysInputDef}
\end{equation}
where $\mathbf{A}$ and $\mathbf{B}$ are real finite linear maps $\mathbb{X}\mapsto\mathbb{X}$ and $\mathbb{U}\mapsto\mathbb{X}$. Suppose $U_t$ is generated by a linear state-space model of the form,
\begin{equation}
	U_t = \mathbf{G}\,U_{t-1} \,+\, U'_t \label{Uasymptoticstationary}
\end{equation}
for a linear $\mathbf{G}:\mathbb{U}\rightarrow\mathbb{U}$. Suppose $U'_t:\Omega\rightarrow\mathbb{U}$ is a stationary process independent of $U_0:\Omega\rightarrow\mathbb{U}$, and both have finite $p\geq1$ moments. For a general starting point, the process $\{U_t\,|\,t\in\mathbb{N}\}$ is not stationary.

\begin{corollary}\label{corollarylinear}
	Consider the map $f:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ of (\ref{recursiveAffineRandSysInputDef}) and a RNN $\widehat{f}$ of the special form (\ref{RNNspecialform}), see also (\ref{genericspecialformRNN}). Let $X_0:\Omega\rightarrow\mathbb{X}$ be random, independent, and have finite $p\geq1$ moments. Assume $\{U_t\,|\, t\in\mathbb{N}\}$ is generated by \eqref{Uasymptoticstationary}. Assume the spectral radius of both $\mathbf{A}$ and $\mathbf{G}$ is strictly less than one. Then for any $\epsilon>0$ there is a finite $\widehat{f}$ and a random initialisation $H_0:\Omega\rightarrow\mathbb{H}^{d_h}$ such that,
\begin{equation}
		\mathsf{E}\big[ \,\big\| \widehat{F}_{t}\cdots\widehat{F}_2\cdot\widehat{F}_1(H_0) \,-\, {F}_{t}\cdots{F}_2\cdot {F}_1(X_0) \big\|^p\, \big]^{{1}/{p}}  ~\leq~ \epsilon, \qquad \forall t\in\mathbb{N}
\end{equation}
\end{corollary}

Proof is given in Appendix \ref{corollarylinearproof}. More general settings are possible; i.e. accommodating some notion of asymptotic stationarity of the random maps $\{\widehat{F}_t \,|\, t\in\mathbb{N}\}$. We leave this generalisation to the reader.

\section{Discussion}

The result in this article is closely related to an application of the Alekseev-Gr\"obner formula \cite{Iserles2009a} in discrete-time, applied to a process that is in some sense stable. Classical feedforward universal approximations then provide control over the local error in the Alekseev-Gr\"obner formulation. This interpretation is exactly that employed in \cite{HansonRaginsky2020a} for approximation of stable ordinary differential equations with bounded inputs using continuous-time RNNs; the Alekseev-Gr\"obner formula being key in \cite{HansonRaginsky2020a}. This trick is not uncommon. For example, similar steps as given are used to prove the time-uniform boundedness of approximation errors in sequential Monte Carlo methods \cite{DelMoralGuionnet2001a}. Viewing the RNN as a perturbed version of some ground truth dynamical system, the analysis herein reinforces the generally applied notion that exponential stability is robust, i.e. it is respected by small perturbations \cite{Sastry1999a,Iserles2009a}. Exponential ergodicity may also be preserved by small perturbations of Markov chain transition functions \cite{RobertsRosenthalSchwartz1998a}.

One key trick used in the proof of Theorem \ref{theoremuniversaltrajapprox} is the telescopic expansion of the error. When combined with a stability property and control over the local, one-step, map approximation error, the result follows easily. Relevant here is that this approach, unlike Alekseev-Gr\"obner, works in a very general setting; e.g. with discrete-time random dynamical systems with random inputs. A second key insight is that the (average) contractive property that ensures the telescopic sum is convergent is the same property that leads to (asymptotic) stationarity in $\mathbb{X}$ and permits control over all the single-step errors.

This work adds to existing related literature. We consider random dynamical systems on non-compact domains with unbounded random inputs. The modelling used here is amenable to practice in many applications, e.g. time-series modelling. Moreover, the network constructions employed are based on practical deep RNNs with simple architectures; and the proof is straightforward so as to facilitate a practical understanding of its workings. We comment on the problem of finite-horizon trajectory approximation later. Prior related work on universal \textit{time-uniform} trajectory approximation, e.g. \cite{Sandberg1991a,Matthews1993a,MatthewsMoschytz1994a,GrigoryevaOrtega2018b,GrigoryevaOrtega2018a,HansonRaginsky2020a,HartHookDawes2020a,GononOrtega2021a}, is focused on deterministic dynamical systems on compact domains with bounded inputs. The article \cite{GononOrtega2020a} is an exception and accommodates unbounded stochastic inputs in the context of universal approximation with echo state networks (see \cite{Jaeger2001a,MaassNatschlaegerMarkram2002a,Lukosevicius2012a,YildizJaegerKiebel2012a}) which are beyond the focus of this work. Also, the conditions in \cite{GononOrtega2020a} are somewhat opaque when applied to the discrete-time random dynamical systems considered herein. For example, we know that certain average contraction conditions, viz. Lemma \ref{eltonlemma}, are loosely necessary to transfer stationarity from the random input sequence to the induced random process. Moreover, here, the induced process is only asymptotically stationary; and the special case in Corollary \ref{corollarylinear} and the proof of Theorem \ref{theoremuniversaltrajapprox} further imply a generalisation to only asymptotically stationary inputs. The conditions proposed herein thus seem to be the natural conditions to frame this kind of result. The derivations given here seem more direct than in related work.

Different network structures, in continuous-time, with various universal approximation capabilities including in some cases time-uniform approximation are given in \cite{MaassSontag2000a,MaassNatschlaegerMarkram2002a,MaassJoshiSontag2007a}. These networks differ from the deep RNNs considered herein; and the modelling, conditions precedent, and approximation capability studied in those articles differs from the discrete-time dynamical systems-based, state-space formulation, presented in this article. We do not address these models further here.

The result presented here has practical relevance in RNN-based trajectory approximation; e.g. in training, design and applications. For example, the result suggests that RNN training sequence lengths may be short in some cases while applications in test time may run indefinitely, with no unbounded accumulation of approximation error. The sparse feedback structure in the deep RNN architecture in the proof of Theorem \ref{theoremuniversaltrajapprox} offers insight into practical network constructions and may simplify design. For example, the notion of memory seems less critical in controlling long-time approximation errors in the problems mainly considered in this work. In applying RNNs, the main result implies a means for testing certain properties of the underlying process approximated; e.g. if performance degrades continually over long time horizons, then the underlying process may violate the considered hypotheses.

In discrete-time, the articles \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a} consider trajectory approximation on fixed finite time intervals. Again, the basic idea in \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a} is that the initial map approximation error is fixed small enough so that the inevitable accumulation of error over the finite interval remains below a desired threshold. Conversely, in the proof of Theorem \ref{theoremFiniteTimeTrajApprox} in Appendix \ref{appendixFiniteTime}, we construct a network for finite-horizon trajectory approximation for very general (random) dynamical systems with inputs. The network we construct has a feedback structure that is not aimed at recursion in the state variables, but rather at memorising the history of inputs and approximating different composed maps at each time step. The error in this formulation does not necessarily accumulate in time; but rather the network size grows significantly as a function of the time interval. This is a very different approach than considered in \cite{JinNikiforukGupta1995a,SchaeferZimmermann2007a} and is of interest on its own. Returning to the practice of network design, it may be of interest to consider the advantages of feedback structures between those based on state recursions and those based on memorisation. For example, it may be possible to improve performance in some systems when only training data is available and the stability/time-delay/dependency nature of the model is unknown. It may simplify modelling of systems that are marginally average contractive. It may reduce overall network size.

We apply an idealised universal approximation theorem in the space of Lipschitz maps \cite{Eckstein2020a,NeumayerGoujonBohraEtAl2022a}. In practice, learning Lipschitz maps with accurate Lipschitz constants or even constrained Lipschitz constants is hard \cite{HusterChiangChadha2018a,AnilLucasGrosse2019a,CohenHusterCohen2019a} and even estimating the Lipschitz constant of a neural network is generally intractable, see \cite{VirmauxScaman2018a}. Constraining or estimating the Lipschitz constant of a neural network, typically feedforward, has many applications (see \cite{HusterChiangChadha2018a,VirmauxScaman2018a,AnilLucasGrosse2019a,CohenHusterCohen2019a}), and in some sense we provide another.

Finally, in this article we consider a general setting amenable to applications. We consider random dynamical systems with unbounded random inputs on non-compact domains. We provide a stepped-through pedagogical proof that is simple and highlights where problems in long-time approximation may arise. There are natural extensions. There may be relaxations or variations of interest on the type of average contraction condition; noting the importance of Lemma \ref{eltonlemma} is in the establishment of some kind of asymptotic stationarity in the induced process. An average exponential rate of contraction is seemingly necessary, so that the telescopic sum in the proof is convergent. A slower average rate will not yield a convergent sum; which is consistent with the general stability theory of perturbed dynamical systems, see \cite{Sastry1999a,Iserles2009a}, and approximation results in \cite{HansonRaginsky2020a}. We only consider the rectified linear activation function in this article. Generalisations of the activation function as in \cite{HornikStinchcombeWhite1989a,LeshnoLinPinkusEtAl1993a,Eckstein2020a,KidgerLyons2020a} are a direction for possible extension. Different types of nonlinear system may also be considered, e.g. \cite{HansonRaginskySontag2021a}.

\section{Additional Illustrative Examples}

\subsection{Universal Time-Uniform Approximation of the Kalman Filter}

Let $\mathbb{Z}\subseteq\mathbb{R}^{d_z}$. We consider the following discrete-time state-space model,
\begin{align}
	Z_{t}^{\mu_0} \,&=\, \mathbf{G}\,Z_{t-1}^{\mu_0} \,+\, E_t \label{sysLintheory} \\
	U_{t} \,&=\, \mathbf{H}\,Z_{t}^{\mu_0} \,+\, D_t \label{measLintheory}
\end{align}
on $\mathbb{Z}\times\mathbb{U}$ where $\mathbf{G}$ and $\mathbf{H}$ are real matrices, and where\footnote{We denote a Gaussian distribution by $\mathcal{N}(\cdot,\cdot)$ with mean given by the first parameter and covariance by the second.} $Z_{0}\sim\mathcal{N}(\overline{Z}_0,\mathbf{C}_0)=:\mu_0$ where $\mathbf{C}_0$ is a positive-semi-definite covariance matrix. The random variables $E_t\sim\mathcal{N}(0,\mathbf{Q})$ and $D_t\sim\mathcal{N}(0,\mathbf{R})$ on $(E_t,D_t)\in\mathbb{Z}\times\mathbb{U}$ are independent, for all $t\in\mathbb{N}$, and of $Z_0$, with $\mathbf{R}$ positive definite.

The process $Z_{t}$ is an underlying signal of interest that is observed by a sensor which produces the signal $U_t$. The Kalman filter for the model (\ref{sysLintheory}), (\ref{measLintheory}) is given by the two equations,

\noindent\begin{minipage}{.495\textwidth}
\begin{align}
	\overline{Z}_{t} &\,:=\ \mathsf{E}\left[ Z_t \,|\, U_1, \cdots, U_t\right] \nonumber\\
	&\,=\,  \mathbf{G}\overline{Z}_{t-1} \,+\, \mathbf{K}_{t}^*\left(U_{t} - \mathbf{H}\mathbf{G}\overline{Z}_{t-1}\right) \label{kalmanstateestimator} \\
	&\,=\,  \left(\mathbf{G} - \mathbf{K}_{t}^*\mathbf{H}\mathbf{G} \right) \overline{Z}_{t-1} \,+\, \mathbf{K}_{t}^* U_{t} \nonumber ~\\\nonumber
\end{align}
\end{minipage}
\begin{minipage}{.495\textwidth}
\begin{align}
	\mathbf{C}_{t} &\,:=\ \mathsf{Cov}\left[ Z_t \,|\, U_1, \cdots, U_t\right] \nonumber\\
	&\,=\, \mathsf{Cov}[ Z_t  - \widehat{Z}_{t} ] \label{kalmancovestimator} \\
	&\,=\, \left(\mathbf{I} - \mathbf{K}_{t}^*\mathbf{H} \right)\left(\mathbf{G}\mathbf{C}_{t-1}\mathbf{G}^\top + \mathbf{Q}\right) \nonumber ~\\\nonumber
\end{align}
\end{minipage}

\noindent with the optimal Kaman gain matrix,
\begin{equation}
	\mathbf{K}_{t}^* \, =\,  \left(\mathbf{G}\mathbf{C}_{t-1}\mathbf{G}^\top + \mathbf{Q}\right)\mathbf{H}^\top\,\left[\mathbf{R} + \mathbf{H}\left(\mathbf{G}\mathbf{C}_{t-1}\mathbf{G}^\top + \mathbf{Q}\right)\mathbf{H}^\top\right]^{-1} \label{kalmangain1}
\end{equation}

The Kalman filter is time-invariant and closed in the sense that the mapping,
\begin{equation}
	(\overline{Z}_{t},\mathbf{C}_{t})  \,=\, f(\overline{Z}_{t-1}, \mathbf{C}_{t-1},U_t) \,=:\, {F}_{t}(\overline{Z}_{t-1}, \mathbf{C}_{t-1}) \label{kalmanmapping}
\end{equation}
does not depend on $t\in\mathbb{N}$ and depends only on the static model parameters $(\mathbf{G}, \mathbf{H}, \mathbf{Q}, \mathbf{R})$ and the current-time observational input $U_t$, i.e. it also does not depend on any recursive computation of higher-order moments, etc. We define the filter state $X_t:=(\overline{Z}_{t},\mathbf{C}_{t})\in\mathbb{X}$ via a slight abuse of notation.

\begin{corollary} \label{linearneuralFilterTheorem}
	Consider a linear-Gaussian signal and observation model (\ref{sysLintheory}), (\ref{measLintheory}). Assume the spectral radius of $\mathbf{G}$ is strictly less than one. Let $\widehat{f}:\mathbb{X}\times\mathbb{U}\rightarrow\mathbb{X}$ denote a RNN of the special form (\ref{RNNspecialform}), see also (\ref{genericspecialformRNN}), taking as input observations in the ordered sequence $\{{U_t}\,|\,t\in\mathbb{N}\}$. Then for any $\epsilon>0$ there exists a finite $\widehat{f}$ and an initialisation $\widehat{x}_0\in\mathbb{X}$ such that,
\begin{equation}
	 \mathbb{E}\left[\, \left\| \, X_t^{x_0}\, -\, \widehat{F}_{t}\cdots\widehat{F}_2\cdot\widehat{F}_1(\widehat{x}_0)\,\right\|^2\, \right]^{{1}/{2}} \,\leq\,\, \epsilon,~\qquad \forall\,t\in\mathbb{N},~~\forall x_0\in\mathbb{X}
\end{equation}
\end{corollary}

\begin{proof}
This is now a straightforward application of Corollary \ref{corollarylinear} after appropriate notational identifications and use of some basic properties of the optimal Kalman filter. We omit the details. See \cite{BishopBonilla2022}.
\end{proof}

We remark that the observation sequence is neither Markov nor stationary in general, and the hypothesis of the corollary do not call for these conditions.

The topic of recurrent neural networks and universal approximation of general Bayesian filters (including time-uniform results) is explored further in \cite{BishopBonilla2022}, with additional examples and empirical results (also contrasting the time-uniform result of Corollary \ref{linearneuralFilterTheorem} with the finite-horizon result in Theorem \ref{theoremFiniteTimeTrajApprox}).

\subsection{Simple Counterexamples with a Discrete-Time Ornstein–Uhlenbeck Process}

Let $\mathbb{X}=\mathbb{R}$. Consider a very simple scalar system of the form,
\begin{equation}
	X^{\mu_0}_t \,=\, f(X_{t-1},U_t) \,:=\,  \varrho + \alpha(X_{t-1} - \varrho) + U_t \label{ar1true}
\end{equation}
where $\alpha\in\mathbb{R}$ is a deterministic parameter, and $U_t\sim\mathcal{N}(0,\varsigma^2)$, for all $t\in\mathbb{N}$. Let $\mu_0 = \mathcal{N}(\varrho_0,\varsigma_0^2)$. Then $X^{\mu_0}_t\sim\mathcal{N}(\varrho_t,\varsigma_t^2)$ is Gaussian for all $t\in\mathbb{N}$ and,
\begin{align}
		\varrho_t \,=\, (1-\alpha^t)\,\varrho + \alpha^t\,\varrho_0 \qquad\mathrm{and}\qquad \varsigma_t\,=\, \alpha^{2t}\varsigma^2_0 + \frac{1-\alpha^{2t}}{1-\alpha^{2}}\varsigma^2
\end{align}
The hypotheses of Lemma \ref{eltonlemma} and Theorem \ref{theoremuniversaltrajapprox} hold with $|\alpha|<1$ and $\{X^{\mu_0}_t \,|\,t\in\mathbb{N}\}$ is asymptotically stationary with Gaussian invariant measure $\mu_\infty:=\mathcal{N}(\varrho,\varsigma^2/(1-\alpha^2))$. If $|\alpha|>1$ the process is unstable.

Consider the simple feedback RNN of (\ref{RNNspecialform}), see also (\ref{genericspecialformRNN}). Let
\begin{align}
	\widehat{X}^{X_0}_t \,=\, \widehat{f}(\widehat{X}_{t-1},U_t) \,:=\, \max\{0, U_t + \varrho - \alpha\varrho + b + \alpha\,\widehat{X}_{t-1}\} - b + \delta \label{ar1NN}
\end{align}
which is of the form (\ref{RNNspecialform}). The parameter $\delta\in\mathbb{R}$ will be used to introduce an approximation error (noting it is a very simplistic error). With $\delta=0$, picking the constant $b>0$ large enough ensures that for all practical purposes $\widehat{f}=f$, e.g. for all practical realisations of $\omega\in\Omega$ simulated.

In Figure \ref{fig:samplesAR1} we plot sample trajectories for the process (\ref{ar1true}) with $\varrho_0=20$, $\varsigma_0^2=\varsigma^2=1$ and $\alpha\in\{0.99,1,1.001\}$. We plot with each sample trajectory the empirical distribution of the trajectory.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.975\textwidth]{samplesAR1.pdf}
    \caption{Sample trajectories for a range of $\alpha$ values showing the corresponding stability/stationarity. }
    \label{fig:samplesAR1}
\end{figure}

Now we introduce some error $|\delta|>0$ into the approximation (\ref{ar1NN}) and compare trajectories with (\ref{ar1true}). We initialise $\widehat{X}_0={X}_0$ almost surely to emphasise the results of Theorem \ref{theoremuniversaltrajapprox}. We plot in Figure \ref{fig:rmseAR1} the empirical root mean squared error averaged over $5000$ realisations of the random input for all $\delta$ ranging from $0.005$ to $0.1$ in $0.005$ increments. The results are unsurprising and consistent with Theorem \ref{theoremuniversaltrajapprox} and with the theory of perturbed stable/unstable dynamical systems \cite{Sastry1999a,Iserles2009a}.

\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.975\textwidth]{rmseAR1.pdf}
    \caption{Root mean square errors. }
    \label{fig:rmseAR1}
\end{figure}

We have considered the case in which the random maps are stationary and given a counterexample when the sequence is not exponentially $p$-contractive (i.e. when $|\alpha|>1$). To see where things may go wrong when stationarity is violated, consider just the case in which $|\alpha|<1$ and let $U_t = \beta \,U_{t-1} + U'_t$ as in (\ref{Uasymptoticstationary}) with $U_t'\sim\mathcal{N}(0,\varsigma^2)$ for all $t\in\mathbb{N}$. The sequence of random maps is exponentially $p$-contractive for any finite $\beta\in\mathbb{R}$. Consider now the identity approximation of $U_t$ given by $\sigma(U_t + b)-b$, see (\ref{ar1NN}), with some large bias $b>0$. If $|\beta|<1$, then Corollary \ref{corollarylinear} applies as $U_t$ is asymptotically stationary; in particular, there is a $b=b(\varepsilon)>0$ such that $\mathsf{E}[\|U_t - (\sigma(U_t + b)-b)\|]\leq\varepsilon$ for any $\varepsilon>0$. However, with $|\beta|>1$, and if $U_t$ moves to the negative, then there is a time $s= s(b)\in\mathbb{N}$ such that $\forall t>s$ we have $\sigma(U_t + b)-b=-b$. The approximation error for all $t>s$ grows on average and a time-uniform bound is impossible (even though the sequence of random maps is exponentially $p$-contractive).

Stationarity and exponential contractivity seem the two natural conditions under which to consider time-uniform results like Theorem \ref{theoremuniversaltrajapprox}/Corollary \ref{corollarylinear}, at least on unbounded domains with unbounded inputs.

We remark for relevance that the Kalman filter state estimator equation (\ref{kalmanstateestimator}) resembles an Ornstein–Uhlenbeck process with the observation process acting as the random driver. See also \cite{BishopBonilla2022} for empirical inference results with RNNs and Kalman filters, contrasting the time-uniform result of Theorem \ref{theoremuniversaltrajapprox}/Corollary \ref{linearneuralFilterTheorem} with the finite-horizon result of Theorem \ref{theoremFiniteTimeTrajApprox}.

\bibliographystyle{apalike}

redpajama_set_name: RedPajamaArXiv


A retro corduroy mini skirt featuring roomy front pockets for everything you wanna bring (e.g. your phone – priorities priorities). Go get it gurl!
In a size 38 the waist width is 74,5 cm and the length is 38 cm. This product runs a little small in size for a tighter fit. The model is 175 cm and is wearing a size 38.


Welcome to Edifying The Body Ministries where our goal is to do just that; edify the members of the body of Christ. Godly edifying can only take place through "sound doctrine" with God the Father's word of truth being preached in truth according to His will, not ours. Dispensational bible study is the only way God's "word of truth" can be fully understood and appreciated. Today we have a 'fool-proof' way to understand God's word, 2 Timothy 2:15 explains the damaging consequences that a bible student will have when he doesn't fully understand God's plan and purpose for Israel, and His plan and purpose for us as well. By understanding this truth, a bible student can fully understand where and when God has dispensed His grace, in His word, throughout time.
Access Code: 925166# Don't forget to press the # button!
Access Code:925166# Don't forget to press the # button!
*ALSO* Access the "Weekly Phone Bible Studies" page for our current Romans chapters 6, 7, and 8 studies on Sonship, and many others!