text:
Identity Theft Defense Lawyer
Under 18 U.S.C. § 1028, certain types of Identity Theft are punishable by imprisonment of up to 15 years.
With identity theft on the rise and thieves developing more and more sophisticated means to obtain unauthorized access to people's personal information, state and federal law enforcement agencies are on high alert for instances of this white collar crime. Many agencies have developed task forces specifically devoted to uncovering and prosecuting identity theft. From members of criminal organizations to individual actors, anyone accused of identity theft is likely to face vigorous prosecution designed to seek maximum punishment for their alleged crime.
Federal Identity Theft Explained
At the federal level, the primary statute that outlaws identity theft is 18 U.S.C. Section 1028. This law identifies eight primary forms of identity theft, which include:
Producing false identification documents
Selling a false or stolen identification document
Knowingly possessing with intent to illegally use or transfer five or more identification IDs
Using a false identification document to defraud the U.S. Government
Producing, selling, or using equipment to make fake IDs
Producing, selling or using someone else's ID with the intent to commit, aid, or abet, any federal crime or a state felony
Trafficking in fake IDs
Knowingly possessing a false or stolen ID issued by the U.S. Government or a sponsoring entity in order to recognize an event of national significance
Within these broad categories, identity theft can take many forms. Examples range from credit card theft and theft of Social Security numbers and other personal information for the purpose of obtaining financial benefits, all the way to offenses involving the use of someone else's identity to make terror threats or breach homeland security protocols. Regardless of the nature of your alleged offense, we have the experience and resources necessary to fight to keep you out of prison.
Defending Against Identity Theft Charges
At Oberheiden, P.C., you will find experienced criminal defense attorneys, former federal prosecutors, and forensic fraud examiners who have chosen to defend those accused of identity theft and other forms of fraud. We are passionate about protecting our clients' legal rights, and we will stop at nothing in using the justice system to ensure that our clients receive a fair outcome. We have successfully represented individuals facing identity theft, fraud, and other criminal allegations, and we will use our experience on both sides of the courtroom to fight to have your charges dismissed.
Penalties for Identity Theft
The penalties for identity theft – which do include serious prison time – vary depending on the nature of the crime and any other criminal activity involved. The potential punishment for identity theft includes forfeiture of any personal property used in connection with the offense, as well as:
Fines and up to 15 years of imprisonment in cases involving the production or sale of fake or stolen U.S. identification documents (such as a U.S. Passport), birth certificates, driver's licenses, or personal identification cards;
Fines and up to 5 years of imprisonment for production, sale, or use of any false identification document other than those listed in the previous bullet. Note, however, that the maximum prison sentence increases back to 15 years in cases involving five or more fraudulent documents;
Fines and up to 20 years of imprisonment for use of a false ID in connection with a drug trafficking crime or a crime of violence;
Fines and up to 20 years of imprisonment for a second offense under 18 U.S.C. Section 1028;
Fines and up to 30 years of imprisonment in cases involving the use of false identification documents to facilitate acts of domestic or international terrorism.
It is important to note that identity theft cases frequently involve other crimes (such as computer fraud or credit card fraud) as well. If this applies to you, you may be facing numerous charges that could add up to life-changing prison time and insurmountable fines. As a result, if you are under investigation or have been arrested for identity theft, it is important to speak with a defense lawyer as soon as possible. There are steps that can be taken to minimize the consequences of an arrest, but the longer you wait, the more difficult it can become to build the strongest possible defense.
Contact Oberheiden, P.C. Today
The attorneys at Oberheiden, P.C. are available to aggressively fight identity theft charges throughout the country. To get started with a free, confidential case evaluation, call our law offices at (888) 727-5154 or contact us online today.
RICO Crimes


text:
Q: Why do downvotes cost reputation for the voter on Meta? On main sites downvotes are usually used to show that answer is incorrect.
On meta sites downvotes are used to express agreement or disagreement.
On main sites downvotes cost reputation so that people downvote only posts that really worth it as it may give bad feelings to answerer.
But it can't be a reason on Meta as it's explicitly said that votes mean another thing here.
So why downvotes cost reputation on Meta?

A: That FAQ is generally considered to be referring to questions, not answers. We downvote questions to indicate we disagree with the ideas they imply or propose. For example, you're proposing a change to how meta.se should work, and I am going to downvote to indicate that I don't like the idea. Note that question votes do not cost reputation on any SE sites.
We don't usually downvote answers to indicate disagreement; contradictory answers can all be voted up on a discussion question, because there can be several valid points of view. But people will downvote my answer if they think it is factually wrong, and upvote it if what it says is factually correct; those are the same reasons for which answers are upvoted or downvoted on the main sites.

redpajama_set_name: RedPajamaStackExchange


text:
using Amqp;
using Helsenorge.Messaging.ServiceBus.Exceptions;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

namespace Helsenorge.Messaging.ServiceBus
{
    [Obsolete("The class 'ServiceBusHttpClient' is a Microsoft ServiceBus specific class and will be removed in the future.")]
    internal class ServiceBusHttpClient
    {
        private readonly ILogger _logger;
        private readonly SemaphoreSlim _authLock = new SemaphoreSlim(1);

        private const int DefaultStsPort = 9355;
        private const string StsPath = "$STS/OAuth/";

        private readonly IHttpClientFactory _httpClientFactory;
        private readonly Uri _stsEndpointUrl;
        private readonly string _username;
        private readonly string _password;
        private string _token;

        public ServiceBusHttpClient(Address address, ILogger logger) : this(address, logger, new DefaultHttpClientFactory())
        {
        }

        public ServiceBusHttpClient(Address address, ILogger logger, IHttpClientFactory httpClientFactory)
        {
            if (address == null)
            {
                throw new ArgumentNullException(nameof(address));
            }

            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
            _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
            _stsEndpointUrl = new Uri($"{(address.UseSsl ? "https" : "http")}://{address.Host}:{DefaultStsPort}{address.Path}");
            _username = address.User;
            _password = address.Password;
        }

        public async Task RenewLockAsync(string queueName, long sequenceNumber, Guid lockToken, TimeSpan lockTimeout, TimeSpan timeout)
        {
            if (string.IsNullOrEmpty(queueName))
            {
                throw new ArgumentException(nameof(queueName));
            }

            if (sequenceNumber <= 0)
            {
                throw new ArgumentException(nameof(sequenceNumber));
            }

            if (lockToken == Guid.Empty)
            {
                throw new ArgumentException(nameof(lockToken));
            }

            _logger.LogDebug($"Renewing lock for message {queueName}/{sequenceNumber}/{lockToken}");

            var remainingTime = await GetAuthTokenAsync(timeout).ConfigureAwait(false);

            var url = new Uri(_stsEndpointUrl, $"{_stsEndpointUrl.LocalPath}/{queueName}/messages/{sequenceNumber:####}/{lockToken}?timeout={lockTimeout.TotalSeconds:####}");
            using (var client = _httpClientFactory.CreateClient())
            {
                client.Timeout = remainingTime;
                client.DefaultRequestHeaders.Add("Authorization", _token);

                var response = await client.PostAsync(url, new StringContent("")).ConfigureAwait(false);
                if (response.IsSuccessStatusCode)
                {
                    _logger.LogDebug($"Successfully renewed lock for message {queueName}/{sequenceNumber}/{lockToken}");
                    return;
                }
                else
                {
                    _logger.LogError($"Renew lock for message with sequence number {sequenceNumber} and lock token {lockToken} on queue {queueName} failed and returned status code {response.StatusCode}");
                }

                var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
                var message = $"Renew lock endpoint returned {response.StatusCode} status code ({content})";
                _logger.LogDebug(message);

                switch (response.StatusCode)
                {
                    case HttpStatusCode.BadRequest:
                    case HttpStatusCode.Forbidden:
                    case HttpStatusCode.Unauthorized:
                        throw new UnauthorizedException(message);

                    case HttpStatusCode.NotFound:
                        throw new MessageNotFoundException($"Message with sequence number {sequenceNumber} and lock token {lockToken} not found in queue {queueName} ({content})");

                    case HttpStatusCode.Gone:
                        throw new MessageNotFoundException($"Queue {queueName} does not exist ({content})");

                    default:
                        throw new RecoverableServiceBusException(message);
                }
            }
        }

        private async Task<TimeSpan> GetAuthTokenAsync(TimeSpan timeout)
        {
            if (_token != null)
            {
                return timeout;
            }

            var deadlineUtc = DateTime.UtcNow + timeout;

            await _authLock.WaitAsync().ConfigureAwait(false);

            try
            {
                if (_token == null)
                {
                    _token = await GetOAuthTokenFromSts(timeout).ConfigureAwait(false);
                }
            }
            finally
            {
                _authLock.Release();
            }

            var remainingTime = deadlineUtc - DateTime.UtcNow;
            if (remainingTime <= TimeSpan.Zero)
            {
                throw new ServiceBusTimeoutException($"Timeout of {timeout} exceeded while obtaining auth token");
            }

            return remainingTime;
        }

        private async Task<string> GetOAuthTokenFromSts(TimeSpan timeout)
        {
            var requestUri = new Uri(_stsEndpointUrl, StsPath);
            _logger.LogDebug("Retrieving oauth token from STS endpoint {requestUri}", requestUri);

            using (var client = new HttpClient())
            {
                client.Timeout = timeout;
                var response = await client.PostAsync(requestUri, new FormUrlEncodedContent(
                    new List<KeyValuePair<string, string>>
                    {
                        new KeyValuePair<string, string>("grant_type", "authorization_code"),
                        new KeyValuePair<string, string>("client_id", _username),
                        new KeyValuePair<string, string>("client_secret", _password),
                        new KeyValuePair<string, string>("scope", _stsEndpointUrl.AbsoluteUri)
                    })).ConfigureAwait(false);

                var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
                if (response.IsSuccessStatusCode)
                {
                    _logger.LogDebug("Oauth token successfully retrieved from STS endpoint {requestUri}", requestUri);
                    return string.Format(CultureInfo.InvariantCulture, "WRAP access_token=\"{0}\"", content);
                }

                var message = $"Sts auth returned {response.StatusCode} status code ({content})";
                _logger.LogDebug(message);

                switch (response.StatusCode)
                {
                    case HttpStatusCode.BadRequest:
                    case HttpStatusCode.Forbidden:
                    case HttpStatusCode.Unauthorized:
                        throw new UnauthorizedException(message);

                    default:
                        throw new RecoverableServiceBusException(message);
                }
            }
        }
    }

    internal class DefaultHttpClientFactory : IHttpClientFactory
    {
        public HttpClient CreateClient(string name)
        {
            return new HttpClient(new HttpClientHandler
            {
                AllowAutoRedirect = true,
                MaxAutomaticRedirections = 1
            });
        }
    }
}

redpajama_set_name: RedPajamaGithub


text:
White-shouldered triller
Campephagidae
Lalage
Lalage sueurii
The white-shouldered triller (Lalage sueurii ) is a passerine bird belonging to the triller genus Lalage in the cuckoo-shrike family Campephagidae. It is found in Indonesia and East Timor. The white-winged triller (L. tricolor ) of Australia and New Guinea was formerly included in this species but is now treated as a separate species.
It is a fairly small bird, 17 centimetres in length. The bill is grey with a black tip and the legs and feet are black. The male is mainly black above and white below. It has a grey rump, white stripe over the eye, white wing-patches and white on the outer tail-feathers. Females have a similar pattern to the males but are brown instead of black above and have fine black barring on the underparts.
The pied triller is similar but is slightly smaller with a broader stripe above the eye and more white in the wing. The male white-winged triller has no white stripe over the eye.
The song of the white-shouldered triller is a metallic whistling which is sometimes given in flight. It also utters a rapid series of harsh notes.
It occurs in eastern Java, southern Sulawesi, Bali and on the Lesser Sunda Islands. It inhabits open woodland, savannas and agricultural land in dry, lowland areas. It feeds on insects, often swooping down from a perch to catch them on the ground.
Oviparous
Oviparous animals are female animals that lay their eggs, with little or no other embryonic development within the mother. This is the reproductive...
Not a migrant
Animals that do not make seasonal movements and stay in their native home ranges all year round are called not migrants or residents.
Indonesia, Timor-Leste
Indomalayan
Anthropogenic biome
1. White-shouldered triller Wikipedia article - https://en.wikipedia.org/wiki/White-shouldered_triller
2. White-shouldered triller on The IUCN Red List site - https://www.iucnredlist.org/species/22706657/118731125
3. Xeno-canto bird call - https://xeno-canto.org/138461
Black-winged cuckooshrike
Lalage melaschistos
White-winged triller
Lalage tricolor
Polynesian triller
Lalage maculosa
Norfolk triller
Lalage leucopyga leucopyga
Black-and-white triller
Lalage melanoleuca
Réunion cuckooshrike
Lalage newtoni
Long-tailed triller
Lalage leucopyga
Black-browed triller
Lalage atrovirens
Lemur catta
Ceratotherium simum

redpajama_set_name: RedPajamaCommonCrawl


Barbara Stabiner is back on Preferred Company to reach out to listeners and tap into their aura, frequency and vibrations to translate what she sees, hears and feels. Listeners will get a chance to ask Barbara personal questions from relationships, job futures and success. What's going on with Aquarius, Gemini and Libra signs and how have roadblocks affected personal relationships and individual employment? Barbara will tell you. Not only is Barbara a visionary life coach and energy counselor, she is a clairvoyant, which means she's clearly seeing into two worlds and tapping into a person's energy field. Your loved one needs home care and suddenly you find yourself facing a million difficult issues! It can become overwhelming, demanding huge amounts of your time, energy and resources. When that time arrives, call Preferred Home Health Care and Nursing Services, 800 603 CARE (2273) or visit preferredcares.com. Preferred Home Health Care and Nursing Services offers 24 hour live in services to all of New Jersey. Their home health aides and licensed practical nurses can take care of clients from 3 to 103. Call 800 603 CARE (2273) or visit preferredcares.com.


I talk to friends and buyer clients all the time who say that they don't want to waste my time because they're still trying to figure out what they want in a home. My reaction to that gracious sentiment is always the same. You're not wasting my time. In fact, if we visit a home and you don't like it, that's the opposite of a waste of our time.
You see, it's my job to help you. To do that to the best of my ability, I need to understand exactly what you're looking for. The more homes we look at, whether you like them or not, the better idea I'll have of what you're looking for.
The alternative to working with a buyer's agent isn't that great. By going to a home and only working with the listing agent, you're not helping anybody (except for maybe the seller).
A buyer's agent does much more than just show a home. That's why it's important to work with one every step of the way. Not only can we help you find the perfect home, we can help you write a contract and negotiate the best terms on the sale.
If you have any questions for us about buyer's agents, seller's agents, or anything else about real estate, give us a call or send us an email. We look forward to hearing from you soon.


I'm glad to see that the majority of you get between 7 and 8 hours of sleep per night. I fall into that category as well, and sometimes I still feel like I need more! As you may know, I have a difficult sleep schedule because I work on a morning newscast- I have to be at work at 3:30 a.m., I wake up around 1:30 a.m., and I go to bed around 6:00! It's strange, but it's forced me to really make sure I'm getting enough sleep, since I want to be refreshed and energized on the news! And as I've found through doing a little research for this blog- the concept of "Beauty Sleep" really is legit! It's a time when your body can heal and repair itself. Experts on ivillage.com recommend 8 hours, and it'll help you steer clear of those dark circles, puffiness, and overall grogginess that doesn't help anyone look or feel good! Below are some of the best tips for getting quality sleep that I've found (besides getting all 8 hours!), courtesy of this article on wikiHow- check it out for a more complete explanation of these points and more.
Sleep in complete darkness- or as close to it as possible- even a little light in the room can disrupt your circadian rhythm. I use an eye mask to block out excess light.
Avoid Caffiene- this sounds common sense- but for some people, even an afternoon can of pop can hinder your ability to sleep later on.
Lose Weight- Being overweight can increase the risk of sleep apnea- preventing a good, restful sleep!
Remove the clock from view- This will keep you from obsessing over what time it is when you can't sleep- which may stress you out more.
Keep your bed for sleeping- don't watch TV or do work there- it will be harder to relax.
Exercise Regularly- Doing so for at least 30 minutes a day can help you fall asleep, just donn't do it too close to your bedtime.
Put Your Work Away- at least 1 or 2 hours before bed. This will help you unwind and be less tense when you finally do go to bed.
Now if only we could follow these tips all the time! I know first hand that it's tough to really make sure you're getting all the sleep you need, let alone the best quality sleep, but hopefully if we can implement some of these ideas, we'll feel a little mor energized when we wake up!
Aw, good tips! I really need more sleep but I'm lucky to get 7 hours. I'm a college student after all! Gotta love those midnight papers and caffeine addictions.
In your last couple of videos, your looking rather dull... ever since the 'Salon Bronze vs. Sun Love: Tanning Comparison' episode..
Hope its not hurt you too much. Don't bother about those comments etc.
I admire your integrity, courage and honesty. U've done real good..
Grown-ups require somewhere in the range of seven and eight long stretches of rest every night. There are singular contrasts inside this range, in spite of the fact that generally, the level of the populace that genuinely needs under seven hours is very little.Try to move on to this best essay writing service for any kind of academic writing work.


text:
\section{Introduction}

Deep learning neural networks (DNNs) are widely used today in many applications such as image classification, object detection, etc. High throughput and high energy efficiency are two of the pressing demands on DNNs these days. However, current computation mechanisms are not the best choices for DNN in terms of efficiency. As DNN models become increasingly complex, the computation process takes a lot of time and energy~\cite{inference2015performance}. In traditional von Neumann architectures, storage memory is separated from computation operations. To compute a output feature map, we need to read the input feature map and weights from the memory units, send them to the computation module to compute before writing the results back to the memory units. During the whole process, the system spends a large portion of energy and time in data movement~\cite{yang2017designing}. This is made worse with advances in process technology, making the relative distances involved even longer and thus more costly.

{\em Emerging memory technologies} (EMT) including PCRAM, STT-RRAM and FeRAM~\cite{meena2014} promise better density and energy efficiency, especially since many are non-volatile and well suited to the read-mostly applications.
In-memory deep learning using EMT cells, especially in {\em analog} mode, has already demonstrated an order of magnitude better performance density and two orders of magnitude better energy efficiency than the traditional deep learning on the MNIST dataset~\cite{yao2020fully}~\cite{cai2019fully}~\cite{luo2021nc}.
As in-memory deep learning integrates computation with the memory operations~\cite{sebastian2020memory}, the computation results can be directly read from the memory modules using a single instruction.  This is different from traditional deep learning, where the memory operations are executed separately from the computation operations. Computing where data is stored reduces the need to move large amounts of data around frequently. Especially when technology scales and on-chip distances become longer, we can expect substantial savings of time and energy if the emerging memory technology is used in the in-memory deep learning paradigm.

A big challenge of in-memory computing is the instability of EMT cells~\cite{raghavan2013rtn}~\cite{luo2018fpga}. Unlike traditional memory technology, where the data are stored in stable memory cells, data stored in analog mode EMT cells may fluctuate and different values can be output. For instance, suppose we store the weight $w$ in an EMT cell. When we read it from the EMT cell, the output may become $w+\Delta w$ instead of $w$. Here $\Delta w$ is a fluctuating amplitude of that memory cell. Because of this instability, in-memory deep learning using EMT especially in an analog manner may make incorrect classification~\cite{du2020exploring}. This can severely limit its application in the real world.

Another challenge of in-memory deep learning is the ineffectiveness of traditional energy reduction techniques, such as pruning, quantization, etc. Pruning reduces energy consumption by decreasing the number of operations~\cite{yang2017designing}~\cite{DeepCompression}. However, the energy of in-memory deep learning is roughly proportional to the weight values. Pruning usually removes those weights with smaller absolute values, which only contribute to a small portion of the energy consumption. Quantization reduces the energy consumption by decreasing the complexity of operations using low precision data~\cite{wang2019haq}~\cite{wang2021evolutionary}. Unfortunately, in EMT cells, low precision data consumes almost the same amount of energy as that of the high-precision data. Therefore, traditional pruning and quantization technologies are less effective for in-memory deep learning.

In this paper, we solve these problems by proposing three techniques. They can effectively recover the model accuracy loss in in-memory deep learning and minimize energy consumption. Our innovations are:
\begin{itemize}
\item\textbf{Device enhanced Dataset:} We propose augmenting the standard training dataset with device information integrated into the dataset. In this way, the models will also learn the fluctuation patterns of the memory cells. This extra information can also help the optimizer avoid overfitting during training and thus improve the model's accuracy.
\item \textbf{Energy Regularization:} We optimize the energy coefficients of memory cells and the model parameters by adding a new term into the loss function. This new term represents the energy consumption of the model. The optimizers can automatically search for the optimal coefficients and parameters, improving the model accuracy and energy efficiency at the same time.
\item \textbf{Low fluctuation Decomposition:} We decompose the computation into several time steps. This is customized for in-memory deep learning and can compensate for the fluctuation of memory cells. This decomposed computation mechanism can improve the model accuracy and energy efficiency.
\end{itemize}
We prove the effectiveness of all these techniques both theoretically and experimentally. They can substantially improve the model accuracy and energy efficiency of models for in-memory deep learning. We organize the rest of the paper as follows: Section~\ref{s2} gives related work; Section~\ref{s3} presents the background knowledge; Section~\ref{s4} introduces our proposed optimization method; Section~\ref{s5} shows the experiment results; Section~\ref{s6} concludes.

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7.0in] {Figure/Overview.eps}
  \caption{(a) Comparison between the traditional memory cell and the EMT cell; (b) The computation mechanism for traditional deep learning; (c) The computation mechanism for EMT-based in-memory deep learning, where we integrate multiplication and addition into the read operations.}
  \label{f:work_overview}
\end{figure*}

\begin{figure}[!t]
  \centering
  \includegraphics[width=3.5in] {Figure/Baisc.eps}
  \caption{(a) The relationship between the energy consumption of the EMT cell and the value of the weight; (b) The probability distribution of the weight fluctuation, with different energy coefficients $\rho$. The traditional cell is marked in grey color for reference.}
  \label{f:basic}
\end{figure}

\section{Related Work}
\label{s2}

The first category of work to improve the accuracy of the in-memory deep learning model is called \textit{binarized encoding}. The information in each memory cell is digitized into one bit instead of being stored as a full precision number. In other words, the data stored in each memory cell is either $1$ or $0$~\cite{emara2014differential}. Theoretically, the one-bit data is more robust than a high-precision number at the same level of fluctuation.  Several previous works used the one-bit design to compute the binarized neural networks. Sun \textit{et al.} used the single-bit cell to execute the XNOR operations in XNOR-Net~\cite{sun2018xnor}. Chen \textit{et al.}~\cite{chen201865nm} used the single-bit cell to execute the basic operations in binary DNN. Tang \textit{et al.} proposed a customized binary network for the single-bit cell ~\cite{tang2017binary}. However, either the XNOR-Net or the binary neural network has a large accuracy drop compared with the full-precision model~\cite{rastegari2016xnor}. Recently, new progress in this research direction is to store a high-precision weight using a group of single-bit memory cells. For example, Zhu \textit{et al.}~\cite{zhu2019configurable} used single-bit memory cells $\times N$ to store a $N$-bit weight. Such a method can increase the model accuracy because it can increase the effective precision of weights. Compared with the traditional design, it uses more memory cells.

The second category of work to improve the accuracy of the in-memory deep learning model is called \textit{weight scaling}. Theoretically, we can reduce the amplitude of weight fluctuation by scaling up the weight values stored in the memory cell~\cite{peng2019optimizing}. After computation, we scale the result down using the same scaling factor. In the literature, many research works have found other physical ways to reduce the amplitude of weight fluctuation. For example, He \textit{et al.} found that we can reduce the fluctuation amplitude by lowering the operation frequency~\cite{he2019noise}. Chai \textit{et al.} found one material, which has lower fluctuation amplitude than the other types of material~\cite{chai2018impact}. However, these methods demand strict physical conditions. Compared with them, weight scaling is a more general method that can reduce the fluctuation amplitude of memory cells in most conditions~\cite{sorbaro2020optimizing}. However, Choi \textit{et al.} found that although the memory cell using scaled weights showed smaller fluctuation amplitude, it also consumed higher energy consumption~\cite{choi2014random}. Ielmini \textit{et al.} modeled the relationship between the scaling factor, the fluctuation amplitude, and the energy consumption~\cite{ielmini2010resistance}, which could help us to find the optimal scaling factor for in-memory computing computation.

The third category of work to improve the accuracy of the in-memory deep learning model is called \textit{fluctuation compensation}. To alleviate the instability, they first read the memory cell by multiple times and then record the statistical result such as mean and standard deviation~\cite{puglisi2015statistical}. Afterward, they either calibrate the model parameter or the model output directly based on that statistical results~\cite {shim2020two}. This method is also widely used during the memory cell programming stage. For example, Joshi \textit{et al.} compensated the programming fluctuation by tuning the batch normalization layer parameters~\cite{joshi2020accurate}.
Alternatively, Zhang \textit{et al.} compensated the programming fluctuation by offsetting the weight values~\cite{zhang2020reliable}. These methods are effective in a static device environment. If we face a dynamic environment, a more general way is needed. One popular approach is to have many equivalent models running in parallel. Then we calculate the mean of the results. Joksas \textit{et al.} did this by applying the committee machine theory into the in-memory computing devices~\cite{joksas2020committee}. Wan \textit{et al.} optimized this process by running a single model on the same device and reading the memory cells multiple times~\cite{wan2020voltage}. This method can average out the weight fluctuation and get a more stable result.

\section{Preliminaries}
\label{s3}

EMT-based in-memory deep learning can be very efficient~\cite{yao2020fully} because of its analog operation. Fig.~\ref{f:work_overview}(a) shows the difference between the traditional and EMT memory cells. When we read a weight $w$ from the traditional memory cell, the input to the corresponding memory cell is $1$, meaning that the read request to that memory cell is enabled. Afterward, the memory cell returns $w$ as an output. An EMT cell for in-memory deep learning is quite different. When we read the weight $w$, the input to the memory cell is a variable $x$ instead of the fixed data $1$. The memory cell then returns $x\cdot w$ directly, the product of the input signals $x$ and the stored weight $w$. In other words, the EMT cell integrates the multiplication operation into the read operation.

Analog EMT is more efficient than traditional memory not only in multiplication operations but also in addition operations~\cite{pedretti2021memory}. To better explain this, we show how traditional cells and EMT cells execute the {\em multiply-accumulate} (MAC) operation in Fig.~\ref{f:work_overview}(b) and Fig.~\ref{f:work_overview}(c), respectively. For traditional memory cells, we first read weight $w_i$ from the corresponding memory cells. Afterward, the output $w_i$ is multiplied by the activation $x_i$ using a multiplier. Finally, we sum all products $x_i\cdot w_i$ from each of the multipliers together either by a single adder sequentially, or use a tree of adders to perform the sum in parallel. To achieve the same computation using EMT-based in-memory computing, we just need to connect the output of each memory cell to the same port. Physically, the sum of all the memory cell outputs $\sum x_i\cdot w_i$ can be obtained from that port directly. This is also referred to as a {\em current sum}.

\subsection{Challenges of Analog In-memory Deep Learning}

The energy consumption of a EMT cell is much less than that of traditional memory cells when they execute the MAC operation. There are important differences. In traditional memory cells, energy consumption is not related to the weight value that is stored. In analog EMT, it is proportional to the weight value~\cite{wang2020ncpower}, as shown in Fig.~\ref{f:basic}(a). We use a parameter $\rho$ to denote this energy coefficient. This parameter is tunable. We can use this parameter to optimize its energy consumption. Theoretically, a small coefficient $\rho$ can help us  improve the energy efficiency of models.

A big challenge in using the merging memory technology is that regardless of the actual technology, their memory cells do not output stable results~\cite{raghavan2013rtn}. Physically, each memory cell has multiple states, and it changes its state with time randomly. Whenever we read the memory cell, it can be in any of the states. At $l$-th state, the weight value read from the memory cell is $r_l(w,\rho)$, where $w$ is the pre-stored weight and $\rho$ is the energy coefficient. Given an input $x$, the output data becomes $x\cdot r_l(w,\rho)$ instead of $x\cdot w$. In Fig.~\ref{f:basic}(b), we show an example of a memory cell with two states. In this example, each state has a $50$\% probability to show. If the store weight is $w$, and the energy coefficient is $\rho$, the output result can be either $x\cdot r_0(w,\rho)$ or $x\cdot r_1(w,\rho)$ depending on its state, where $x$ is the input activation. The bottom three sub-graphs in Fig.~\ref{f:basic}(b) corresponds to memory cells with three different energy coefficients $\rho$.

The fluctuation shown in Fig.~\ref{f:basic}(b) is a simplified example. In practical EMT cells, the number of fluctuation states and the probability of each state are more complicated. There are many works in the literature studying these fluctuations~\cite{gong2018signal}.
For deep learning, such a phenomenon can cause a non-negligible accuracy drop. It is the biggest challenge that limits the application of in-memory computing. As we can see from Fig.~\ref{f:basic}(b), the fluctuation amplitude, defined as the average distance among $r_l(w,\rho)$, is related to the energy coefficient $\rho$. Theoretically, a higher $\rho$ will result in less fluctuation of the weight and thus higher model accuracy, but it also means higher energy consumption of memory cells. This trade off is the focus of this paper.

\subsection{Incompatibility of Traditional Training Method}
\label{s:risk}

Fig.~\ref{f:overview_trad} shows the standard training process in deep learning. The loss function takes the dataset and the model and generates a measure of the distance between the current parameter values and their optimal values. The optimizer uses gradient descent to reduce this distance by updating the parameters. This process would iterate for many epochs until the optimizer can find an optimal set of parameters. We define $\mathbf{X}\in \mathcal{X} $ as the image data and $\mathbf{Y}\in \mathcal{Y} $ as the label data. Here $\mathcal{X}$ and $\mathcal{Y}$ denote the spaces for images $\mathbf{X}$ and labels $\mathbf{Y}$. For simplicity, we express a one-layer neural network model in Equation~(\ref{e:whole}), where the weight $\mathbf{W}$ and the bias $\mathbf{B}$ are both trainable parameters.
\begin{equation}
\mathbf{Y}  = f(\mathbf{X}) = \mathbf{W}\mathbf{X}+\mathbf{B}
\label{e:whole}
\end{equation}
Let's define the function class $\mathcal{F}\subset \mathcal{Y}^\mathcal{X}$ as the search space of function $f$. Let $ \mathcal{Z} = \mathcal{X} \times \mathcal{Y}$ be the combination of $\mathcal{X}$ and $\mathcal{Y}$, and $\textbf{Z}=(\textbf{X},\textbf{Y})$ be the combination of $\textbf{X}$ and $\textbf{Y}$. Define $\mathfrak{D}$ as the unknown distribution of data $\textbf{Z}$ on space $\mathcal{Z}$. Given the above definitions, the loss function can be expressed as $\mathcal{L}:\mathcal{F} \times \mathcal{Z} \rightarrow \mathbb{R}$. It is the mapping from the combination of $\mathcal{F}$ and $\mathcal{Z}$ to the real number space $\mathbb{R}$. Theoretically, the training process of the neural network is to find the optimal function $f$ from the function class $\mathcal{F}$ that can minimize the risk, i.e., the expectation of the loss function. We express it in Equation~(\ref{e:risk_old}).
\begin{equation}
R[{f}]=\mathbb{E}_{{z}\sim {\mathfrak{D}}}\big[\mathcal{L}({f},{z})\big]
\label{e:risk_old}
\end{equation}

\begin{figure}[!t]
  \centering
  \includegraphics[width=3.5in] {Figure/MethodsOld.eps}
  \caption{The training method for traditional deep learning. The dataset (green blocks) includes images and labels.}
  \label{f:overview_trad}
\end{figure}

The difficulty in solving this problem is that the distribution $\mathfrak{D}$ is unknown. What we have is just the dataset $\{\textbf{Z}_1,\cdots,\textbf{Z}_N\}$, which are {\em independent and identically distributed} (i.i.d.) samples from the distribution $\mathfrak{D}$. Alternatively, the traditional training process of the neural network is to find the optimal model $f$ from the function class $\mathcal{F}$ that can minimize the empirical risk, i.e., the average of loss functions on the sampled dataset. We express it in Equation~(\ref{e:risk_oldd_sample}).
\begin{equation}
R_s[{f}]=\frac{1}{N}\sum_{i=1}^{N}{\mathcal{L}({f},{z_i})}
\label{e:risk_oldd_sample}
\end{equation}
The distance between the risk and the empirical risk is called the {\em generalization error} $\epsilon$, expressed in Equation~(\ref{e:risk_old_delta}). The crucial problem of the training process is to make sure that the generalization error $\epsilon$ can be bounded.
\begin{equation}
\epsilon= R[{f}] - R_s[{f}]
\label{e:risk_old_delta}
\end{equation}

Researchers have since solved this math problem for traditional deep learning models. Many theoretical studies have shown that neural network optimizers, such as {\em stochastic gradient descent} (SGD)~\cite{bottou2012stochastic} or {\em adaptive moment estimation} (Adam)~\cite{zhang2018improved} can efficiently find the optimized function $f$. However, for in-memory deep learning models, because of the fluctuation of weight matrix $\mathbf{\widetilde{W}}$, the sampled function $f$ becomes $\widetilde{f}$ and the distance of risks becomes $\widetilde{\epsilon}$, as expressed in Equation~(\ref{e:risk_old_delta_2}). Therefore, the traditional training method no longer works for this problem. We need a new training method that is suitable for in-memory deep learning models.
\begin{equation}
\widetilde{\epsilon} = R[{f}] - R_s[\widetilde{f}]
\label{e:risk_old_delta_2}
\end{equation}

\begin{figure}[!t]
  \centering
  \includegraphics[width=3.5in] {Figure/Methods.eps}
  \caption{The proposed training method for in-memory deep learning. The dataset (green blocks) includes extra fluctuation data. We mark the proposed three optimization techniques in red color.}
  \label{f:overview}
\end{figure}

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7in] {Figure/Train_1.eps}
  \caption{(a)-(f) Visualization of the device-enhanced datasets. Pixels forming letters \texttt{A} and \texttt{B} denote the image data labeled by class A and class B, respectively. The blurred pixels in (e) and (f) indicate the distribution of images $\mathbf{X}$ and fluctuation $\mathbf{S}$.}
  \label{f:process_path1}
\end{figure*}

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7in] {Figure/Train_2.eps}
  \caption{(a)-(c) Visualization of the traditional datasets. The orange straight lines in (a) and (b) indicate the absence of device information.}
  \label{f:process_path2}
\end{figure*}

\section{Optimizing for In-memory Deep Learning}
\label{s4}

Our training method for in-memory computing can effectively improve the model accuracy, with improved energy efficiency. Fig.~\ref{f:overview} shows the overview picture. Essentially, we propose three optimization techniques and integrate them into the training process. The first technique is called {\em device-enhanced datasets}. This technique integrates device information as additional data into the dataset. This will make the model more robust to the fluctuations of the device. The second technique is called {\em energy regularization} which we add a new regularization term into the loss function that makes the optimizer reduce the energy consumption automatically. The third technique is called {\em low-fluctuation decomposition} by which we decompose the computation involved into several time steps. This decomposition achieves high model accuracy and energy efficiency. We shall now give the mathematical basis of these three techniques.

\subsection{Device-Enhanced Dataset}

Our first optimization technique is to enhance the dataset with device information. In addition to the regular image data $\mathbf{X}$ and the label data $\mathbf{Y}$, our enhanced dataset has another source of data, the fluctuation $\mathbf{S}$, which reflects the random behavior of memory cells. Fig.~\ref{f:process_path1} shows an visualized example of $\mathbf{X}$, $\mathbf{Y}$ and $\mathbf{S}$, with four training data (Fig.~\ref{f:process_path1}(a))-(Fig.~\ref{f:process_path1}(d)). They can be classified as either letter \texttt{A} or letter \texttt{B}. Images in the same class can have different variants. Take the letter \texttt{A} for instance. It can be in any font, either normal (Fig.~\ref{f:process_path1}(a)) or italic (Fig.~\ref{f:process_path1}(c)). No matter what the variant is, its pixels must follow $\mathfrak{D}_\texttt{A}$, the distribution for class \texttt{A}. After training, we can accurately classify any image that belongs to class \texttt{A} as long as its pixels follow the distribution $\mathfrak{D}_A$ (Fig.~\ref{f:process_path1}(e)). This truth also holds for images of letter \texttt{B}, whose pixels follow distribution $\mathfrak{D}_\texttt{B}$. The fluctuation data $\mathbf{S}$ reflects the random states of memory cells. In our visualized example, the pixels indicate the states of the memory cells. The patterns for fluctuation $\mathbf{S}$ also follow a certain distribution $\mathfrak{R}$, which can be learned during training. Using this enhanced dataset, the model can make correct predictions for in-memory deep learning because now it becomes aware of the fluctuations (Fig.~\ref{f:process_path1}(e))-(Fig.~\ref{f:process_path1}(f)).

As we integrate the fluctuation device data into the dataset, the model will not overfit during training. In Fig.~\ref{f:process_path2}, we visualized an example of training using only the dataset $\mathbf{X}$ and do not consider the device information. All pixels in the data $\mathbf{S}$ are in the center of the matrix, indicating the absence of device information (Fig.~\ref{f:process_path2}(a))-(Fig.~\ref{f:process_path2}(b)). During training, the model will overfit this static data $\mathbf{S}$ because it does not have any variant. As we can see from Fig.~\ref{f:process_path2}(c), the distribution learned by the model is only a straight line (orange), which is different from the real fluctuation of memory cells (purple). Therefore, the model will mis-classify the images. On the other hand, if we include the fluctuation data $\mathbf{S}$, the overfit can be avoided. As we can see from Fig.~\ref{f:process_path2}(e) and Fig.~\ref{f:process_path2}(f), the fluctuation of memory cells (purple) will follow the learned distribution (orange) so that the model can accurately classify the images.

We developed a method to integrate the fluctuation data $\mathbf{S}$ into the training process. To simplified this problem, we first decompose the computation of neural network model (Equation~(\ref{e:whole})) into several sub-tasks. For example, each element $y_{ij}$ in the output matrix $\mathbf{Y}$ can be computed independently using Equation~(\ref{e:workload}). Here vector $\mathbf{w}_i$ is the $i$-th row in weight matrix $\mathbf{W}$, vector $\mathbf{x}_j$ is the $j$-th column in the input matrix $\mathbf{X}$, and $b_{ij}$ is the element in the bias matrix $\mathbf{B}$ at $i$-th row and $j$-th column.
\begin{equation}
y_{ij}  = \mathbf{w}_i\mathbf{x}_j+b_{ij}
\label{e:workload}
\end{equation}
As we showed in Fig.~\ref{f:basic}, the weight we read from the EMT cell is unpredictable. Physically, the memory cell changes its status randomly, and the exact output value depends on the state of the memory cell when we are reading it. We denote $w_{ik}$ be the $k$-th element in vector $\mathbf{w}_i$. $\widetilde{w}_{ik}(j)$ is the sampled data when we read $w_{ik}$ from the memory cell, and multiply it with the input vector $\mathbf{x}_j$. Mathematically, we can express $\widetilde{w}_{ik}(j)$ as a polynomial, as shown in Equation~(\ref{e:combination}).
\begin{equation}
\widetilde{w}_{ik}(j) = \sum_{l=1}^m{r_l(w_{ik}, \rho)\cdot s_{ijkl}}
\label{e:combination}
\end{equation}
We use $r_l(w_{ik},\rho)$ to denote the weight retrieved when the memory cell is at $l$-th state. It can be considered as a function of the pre-stored weight $w_{ik}$ and energy coefficient $\rho$. At any moment, each memory cell can only be in one state. Hence, we use a one-hot encoded vector [$s_{ijkl}$]$_{1\leq l\leq m}$ to indicate the state of the memory cell when $w_{ik}$ is sampled. The value of $s_{ijkl}$ is shown in Equation~(\ref{e:coeff}). For given indexes $i$, $j$, and $k$, if the corresponding memory cell is at $l_{0}$-th state, only $s_{ijkl_0}$ equals $1$ and all the other coefficients equal $0$.
\begin{equation}
s_{ijkl}=
\begin{cases}
    1  ~& \text{if } l = l_{0}\\
    0  ~& \text{if } l \neq l_{0}
\end{cases}
\label{e:coeff}
\end{equation}

As we can see from Equation~(\ref{e:combination}), the sampled weight $\widetilde{w}_{ij}$ from the memory cell consists of two parts:

\begin{itemize}
\item The {{\textit{deterministic}}} parameter $r_l(w_{ik},\rho)$ is a function indicating the returned value from the memory cell storing weight $w_{ik}$ for a memory cell that is in the $l$-th state. We denote the matrix $[r_l(w_{ik}),\rho]$, as $\mathbf{r}(\mathbf{w}_i,\rho)$, shown in Equation~(\ref{e:matrix1}). $\mathbf{r}(\mathbf{w_i},\rho)$ can be considered a function of the weight vector $\mathbf{{w}}_i$ and the energy coefficient $\rho$.

\item The {{\textit{stochastic}}} parameter $s_{ijkl}$ is a random coefficient indicating whether the memory cell storing weight $w_{ik}$ is at $l$-th state when it is sampled and multiplied with the input vector $\mathbf{x}_j$. We denote matrix $[s_{ijkl}]$ as $\mathbf{S}_{ij}$, shown in Equation~(\ref{e:matrix2}). $\mathbf{S}_{ij}$ can be considered as a part of fluctuation data $\mathbf{S}$.
\end{itemize}
\begin{align}
\mathbf{r}(\mathbf{w}_i,\rho) =&
\begin{bmatrix}
r_1(w_{i1},\rho) &r_1(w_{i2},\rho) &\cdots &r_{1}(w_{in},\rho) \\
r_2(w_{i1},\rho) &r_2(w_{i2},\rho) &\cdots &r_{2}(w_{1n},\rho) \\
\vdots  &\vdots  &\ddots &\vdots  \\
r_m(w_{i1},\rho) &r_m(w_{i2},\rho) &\cdots &r_{m}(w_{in},\rho)
\end{bmatrix}\label{e:matrix1}\\
\mathbf{S}_{ij} = &
\begin{bmatrix}
s_{ij11} &s_{ij21} &\cdots &s_{ijn1} \\
s_{ij12} &s_{ij22} &\cdots &s_{ijn2} \\
\vdots  &\vdots  &\ddots &\vdots  \\
s_{ij1m} &s_{ij2m} &\cdots &s_{ijnm}
\end{bmatrix}
\label{e:matrix2}
\end{align}
We can now integrate the fluctuation data $\mathbf{S}$ into the training process for in-memory deep learning. Each element $y_{ij}$ in the output matrix $\mathbf{Y}$ can be calculated using Equation~(\ref{e:sample}). $y_{ij}$ is a function of both the deterministic and stochastic parameters. Here $\mathbf{\widetilde{w}}_i$ refers to the sampled weight vector read from the memory cells. For simplicity, we assume the bias $b_{ij}$ is a deterministic parameter. In some cases, the bias is also fluctuating. We can use the same method to separate deterministic and stochastic parameters for $b_{ij}$.
\begin{equation}
y_{ij}  = \mathbf{\widetilde{w}}_i\mathbf{x}_j + b_{ij} =\mathbf{1}\big(\mathbf{r}(\mathbf{w}_i,\rho)\circ \mathbf{S}_{ij}\big)\mathbf{x}_j + b_{ij}
\label{e:sample}
\end{equation}
The $\circ$ operator between $r(\mathbf{w}_i, \rho)$ and $\mathbf{S_{ij}}$ is the {\em Hadamard product}, i.e., element-wise product. The unit vector $\mathbf{1}$ is expressed in Equation~(\ref{e:unit}). We use it to sum up the entire column of the target matrix.
\begin{equation}
\mathbf{1} =
{\small \underbrace{\begin{bmatrix}
1 &1  &\cdots &1 \\
\end{bmatrix}}_{m}}
\label{e:unit}
\end{equation}

\subsection{Energy Regularization}

\begin{figure}[!t]
  \centering
  \includegraphics[width=3.5in] {Figure/EnergyMove.eps}
  \caption{Energy-saving from the energy regularization term. It can decrease both the energy coefficient $\rho$ and the mean of weight values $\overline{w}$.}
  \label{f:energy_term}
\end{figure}

Our second optimization technique adds an energy regularization term into the loss functions during training. From Equation~(\ref{e:sample}) we can infer that the loss function of the model $\mathcal{L}_0$ is a function of weights $\mathbf{w}$ and energy coefficient $\rho$. The target of our optimization technique is to find the optimal energy coefficient $\rho$ that can improve both the model accuracy and energy efficiency. However, it is not an easy task. We prefer a smaller $\rho$ for higher energy efficiency. However, as we see in Fig.~\ref{f:basic}, the higher $\rho$ causes a larger fluctuation amplitude of the weights, which results in accuracy loss. On the other hand, if we choose a larger coefficient $\rho$, the model accuracy would be less affected by the weight fluctuation, but the energy consumption becomes larger.

Our new loss function $\mathcal{L}$ is expressed in Equation~(\ref{e:energy_reg}). The first term $\mathcal{L}_0$ is the original loss function of the model, and the second term represents the energy consumption of the model. $\lambda$ is a hyper-parameter indicating the significance of the energy regularization term. $\alpha_t$ is a constant indicating the number of reading operations from the memory cell
storing weight $w_t$. The overall loss function $\mathcal{L}$ can be considered as a function of $\textbf{w}$ ($w_t$ is the $t$-th element of $\textbf{w}$) and $\rho$, which are both trainable parameters. We can use any popular optimizer (such as SGD optimizer~\cite{bottou2012stochastic} or Adam optimizer~\cite{zhang2018improved}) to search for the optimal weight $\mathbf{w}$ and energy coefficient $\rho$.
\begin{equation}
\mathcal{L}(\textbf{w},\rho)= \mathcal{L}_0(\textbf{w}, \rho) + \lambda \sum_{t}{\alpha_t \rho |w_t|}
\label{e:energy_reg}
\end{equation}

During training, gradient descent will minimize the loss function $\mathcal{L}$. After optimization, both $\rho$ and $w_t$ will become smaller. We show this process in Fig.~\ref{f:energy_term}. With the help of the energy regularization term, we can improve both the model accuracy and energy efficiency simultaneously.

\subsection{Low-fluctuation Decomposition}

The third optimization technique is to decompose the computation process into multiple time steps. We can visualize the computation involved in Fig.~\ref{f:coding}. The input activation $x$ and the weight $w$ equals the length of the horizontal bar and the vertical bar, respectively. The computation result got from the memory cell equals the area of the square, whose two edges have the same length as the horizontal bar and the vertical bar. In the example of original computing (Fig.~\ref{f:coding}(a)), the lengths of the horizontal bar and vertical bar are
seven ($x=7$) and one ($w=1$), respectively. The area of the output square is thus seven ($x\cdot w=7$).

Theoretically, we can express any input $x$ as a polynomial, as shown in Equation~(\ref{e:decom}). Here the fraction bit $\delta_p$ is binary data, which equals either $0$ or $1$, and $2^p$ is the scaling factor for that term. For example, if the input equals $7$, we can decompose it into three parts: $1\cdot 2^0$, $1\cdot 2^1$, and $1\cdot 2^2$.
\begin{equation}
x= \sum{(\delta_p\cdot 2^p)}
\label{e:decom}
\end{equation}
In our low-fluctuation decomposed computation mechanism (Fig.~\ref{f:coding}(b)), we read each memory cell in multiple time steps instead of once. As Equation~(\ref{e:accum}) shows, at each time step, we only input the fraction bit $\delta_p$ to the memory cell, obtain the value of $\delta_p\cdot w$, and then scale the output from the memory cell by the factor $2^p$. Finally, we sum up all the results from each time step. In this example,  we use three steps to process input seven ($x=7$). The scaling factor of each time step is $2^0$, $2^1$, and $2^2$, respectively. As the weight is one ($w=1$), the final accumulated result is seven ($x\cdot w=7$), the same result as the original computing mechanism.
\begin{equation}
x\cdot w= \sum{(\delta_p\cdot w \cdot 2^p)}
\label{e:accum}
\end{equation}

\begin{figure}[!t]
  \centering
  \includegraphics[width=3.5in] {Figure/coding.eps}
  \caption{(a) Original computation mechanism; (b) low-fluctuation decomposed computation mechanism. The length of the bar denotes activation/weight value. The area of the square denotes their product. The solid/hollow yellow block represents positive/negative fluctuation.}
  \label{f:coding}
\end{figure}

As the name indicates, our low-fluctuation decomposition can alleviate the fluctuations of the memory cell effectively. We can explain this using Fig.~\ref{f:coding}, where we show the fluctuation amplitude in the yellow blocks. The block and hollow block denote positive and negative fluctuation amplitudes, respectively. As we can see from Fig.~\ref{f:coding}(b), using the decomposed computation mechanism,  the negative fluctuation amplitude (hollow block) at the third time step can partially average out the positive fluctuation amplitude at the second time step (solid block). Statistically, the accumulated fluctuations from the decomposed computation mechanism have a lower standard deviation than that of the original computation mechanism.

We can mathematically compare their standard deviations. Equation~(\ref{e:o_trad}) shows the standard deviation of the original computation mechanism, where $O_{\text{ori}}$ is the original output, and $\sigma(w)$ is the standard deviation of $w$ when we read it from the memory cell. Equation~(\ref{e:o_new}) shows the standard deviation of our low-fluctuation decomposed computation mechanism, where $O_{\text{new}}$ is the new output, and $w(p)$ is the weight sampled from the memory cell at $p$-th time step. Since reading memory cells can be considered as independent events, we have $\sigma(w(p)) = \sigma(w)$.
\begin{align}
\ \sigma(O_{\text{ori}}) =\sigma(x\cdot w) = \sum{2^{p}\delta_p}\cdot \sigma(w)
\label{e:o_trad}
\end{align}
\begin{equation}
\begin{aligned}
\sigma(O_{\text{new}})= \sigma\big(\sum{2^{p}\delta_pw(p)}\big) = &\sqrt{\sum{2^{2p}\delta^2_p\sigma^2\big (w(p)\big )}} \\
=& \sqrt{\sum{2^{2p}\delta^2_p}}\cdot \sigma(w)\\
<& \sqrt{(\sum{2^{p}\delta_p})^2}\cdot \sigma(w)
\label{e:o_new}
\end{aligned}
\end{equation}
We can infer from Equation~(\ref{e:o_trad}) and~(\ref{e:o_new}) that our decomposed computation result has a lower standard deviation than the original computation result, leading to high model accuracy (Equation~(\ref{e:o_compare})).
\begin{align}
\sigma(O_{\text{new}}) < \sigma(O_{ori})
\label{e:o_compare}
\end{align}

Our low-fluctuation decomposition can also improve energy efficiency. To prove this, we express the energy consumption of the two computation mechanisms in Equation~(\ref{e:e_list}). From the equation, we can infer that our decomposed computation mechanism consumes less energy than the original one (Equation~(\ref{e:e_compare})).
\begin{align}
&E(O_{\text{ori}}) = \rho\cdot x \ \ \ \ \ \  E(O_{\text{new}}) = \rho \sum \delta_p \label{e:e_list} \\
&E(O_{\text{new}}) < E(O_{\text{ori}})
\label{e:e_compare}
\end{align}

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7in] {Figure/ListAcc.eps}
  \caption{The comparison between our proposed optimization solutions and the traditional optimizer. We test models on the CIFAR-10 dataset. The first row of sub-figures shows the accuracies in a zoomed range, and the second row of sub-figures shows accuracies in the full range.}
  \label{f:performance}
\end{figure*}

\subsection{Convergence of the Training Method}

We shall now mathematically prove the convergence of our training method.
Equation~(\ref{e:whole2}) shows the basic relationship between image data $\mathbf{X}$ and label data $\mathbf{Y}$ in the traditional deep learning network. The output matrix $\mathbf{Y}\in \mathcal{Y}$ is a function $f$ of the input matrix $\mathbf{X}\in \mathcal{X}$. Here $\mathcal{X}$ and $\mathcal{Y}$ denote the space of $\mathbf{X}$ and $\mathbf{Y}$, respectively.
\begin{equation}
\mathbf{Y}  = f(\mathbf{X}
\label{e:whole2}
\end{equation}
For in-memory deep learning applications, the computation becomes unpredictable because of weight fluctuation. As can be seen from Equation~(\ref{e:sample}), output $y_{ij}$ is a function of both the input $\mathbf{x}_j$ and the fluctuation data $\mathbf{S}_{ij}$. To generalize this, the output matrix $\mathbf{Y}\in \mathcal{Y}$ can be defined a function $\widetilde{f}$ of the input matrix $\mathbf{X}\in \mathcal{X}$ and the fluctuation data $\mathbf{S}\in \mathcal{S}$, shown in Equation~(\ref{e:PIM}). Here $\mathcal{S}$ denotes the space of $\mathbf{S}$.
\begin{equation}
\mathbf{Y} = \widetilde{f}(\mathbf{X, S})
\label{e:PIM}
\end{equation}
We define a new data $\widetilde{\mathbf{Z}}\in \widetilde{\mathcal{Z}}$ as the combination of images $\mathbf{X}$, labels $\mathbf{Y}$ and fluctuations $\mathbf{S}$.
Here the space $\widetilde{\mathcal{Z}}$ is the combination of $\mathcal{X}$, $\mathcal{Y}$, and  $\mathcal{S}$. Since  $\mathcal{Z}$ is the combination of $\mathcal{X}$ and $\mathcal{Y}$, we can express the space $\widetilde{\mathcal{Z}}$ as the combination of the spaces $\mathcal{Z}$ and $\mathcal{S}$. We show their relationships in Equation~(\ref{e:parameter1})-(\ref{e:parameter2}).
\begin{align}
\widetilde{\mathbf{Z}}=&(\mathbf{X}, \mathbf{Y}, \mathbf{S})\label{e:parameter1}\\
\widetilde{\mathcal{Z}}=&\mathcal{X}\times \mathcal{Y}\times\mathcal{S}= \mathcal{Z}\times \mathcal{S}
\label{e:parameter2}
\end{align}
Given the fact that $\mathcal{Z}$ follows distribution $\mathfrak{D}$, while $\mathcal{S}$ follows distribution $\mathfrak{R}$, we can infer that the space $\widetilde{\mathcal{Z}}$ (the combination of $\mathcal{Z}$ and $\mathcal{S}$) must follow a distribution $\widetilde{\mathfrak{D}}$ (the combination of $\mathfrak{D}$ and $\mathfrak{R}$). We show this in Equation~(\ref{e:new_dist}).
\begin{equation}
       \mathcal{Z}\sim\mathfrak{D}, \mathcal{S}\sim\mathfrak{R}
\ \ \ \Rightarrow
\ \ \ \widetilde{\mathcal{Z}}\sim \widetilde{\mathfrak{D}}
\label{e:new_dist}
\end{equation}

Finally, our proposed training process for in-memory deep learning models can be concluded as follows: given a function $\widetilde{f}$ in the space $\widetilde{\mathcal{F}}$, and a loss function $\mathcal{L}:\widetilde{\mathcal{F}} \times \widetilde{\mathcal{Z}}\rightarrow \mathbb{R}$, we would like to find $\widetilde{f}\in \widetilde{\mathcal{F}}$ that can minimize the risk, \textit{i.e.}, the expectation of the loss function (Equation~(\ref{e:risk})).
\begin{equation}
R[\widetilde{f}]=\mathbb{E}_{\widetilde{z}\sim \widetilde{\mathfrak{D}}}\big[\mathcal{L}(\widetilde{f},\widetilde{z})\big]
\label{e:risk}
\end{equation}

Thus, we convert the convergence problem of the new training process for in-memory computing into the convergence problem for regular neural networks (see Section~\ref{s:risk}). Finding the optimal function is well studied, and we can use various existing optimizers, such as SGD or Adam, to find the optimal function $\widetilde{f}$.

\section{Experiments}
\label{s5}

We trained the models on the Pytorch platform and evaluate the energy consumption in an in-memory deep learning simulation platform~\cite{lee2019system}~\cite{wang2020ncpower}. To accelerate the training process, we start each experiment from a well-trained model with full-precision weights~\cite{pytorchzoo} and then fine-tune the model by applying our proposed optimizations. During fine-tuning, we quantize both the activations and weights. To form the device-enhance dataset, we fetch the images and labels from the regular datasets (such as CIFAR-10 and ImageNet) as data $\textbf{X}$ and data $\textbf{Y}$, respectively. The fluctuation data $\textbf{S}$ are obtained from state-of-the-art device models~\cite{ielmini2010resistance}. We use a workstation with an Nvidia 2080TI graphic card to train the model. For the CIFAR-10/ImageNet dataset, each experiment can finish in about an hour/an entire day.

We proposed three solutions, denoted as `A',  `A+B', and `A+B+C'. As shown in Fig.~\ref{f:overview}, the notations A, B, C stand for the device-enhanced dataset, energy regularization, and low-fluctuation decomposition, respectively. Solution A uses only the first technique; solution A+B applies the first two; solution A+B+C combines all three. We evaluate popular models including VGG-16, ResNet-18/34 and MobileNet. VGG-16 is a regular deep neural network with only $3\times 3$ kernels. ResNet-18/34 are popular models that achieve competitive accuracy by adding residual links between layers. MobileNet is a small-size model that achieves high efficiency due to its special depthwise layer. We compare our work with three state-of-the-art solutions: binarized encoding~\cite{zhu2019configurable}, weight scaling~\cite{ielmini2010resistance}, and fluctuation compensation~\cite{wan2020voltage} as described in detail in Section~\ref{s2}.

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7in] {Figure/RobInfo.eps}
  \caption{The energy comparison between our proposed solutions and the state-of-the-art, under three levels of fluctuation intensity. We test models on the ImageNet dataset. Both our solutions and the state-of-the-art are free to tune the energy coefficient $\rho$.}
  \label{f:rob}
\end{figure*}

\begin{figure*}[!t]
  \centering
  \includegraphics[width=7in] {Figure/SOTA.eps}
  \caption{The accuracy comparison between our proposed solutions and the state-of-the-art. We test models on the ImageNet dataset. The solid bar and hollow bar denote the top-1 accuracy and top-5 accuracy, respectively. The dash line denotes baseline model accuracy on GPU.}
  \label{f:sota}
\end{figure*}

\subsection{Ablation Study of Proposed Techniques}

Models optimized by our proposed methods have much higher accuracy than the model trained by the traditional optimizer. In Fig.~\ref{f:performance}, we show the accuracy achieved by solution A, solution A+B, and solution A+B+C under different energy budgets. As a reference, we also give the model accuracy trained by the traditional optimizer. As we can see from the figure, at 16 $\mu$J energy budget, the accuracy of solution A+B+C is very close to baseline accuracy (shown as dashed lines in the top sub-figures). On the other hand, the traditional optimizer exhibits relatively low accuracy due to its unawareness of memory fluctuation.

We can see that when the energy budget is decreased, models trained using the traditional optimizer show a dramatic decrease in accuracy. On the contrary, our solution A+B+C can achieve high model accuracy even if we reduce the energy budget. Even just using Solutions A and A+B is enough to maintain a relatively high accuracy, only to be outperformed by Solution A+B+C. This observation further proves the effectiveness of our proposed three techniques on in-memory computing.

We can also see that under 16 $\mu$J energy consumption, the ResNet-18 trained by the traditional optimizer shows much lower accuracy than the VGG-16. By using our solution A+B+C, ResNet-18 can fully recover the accuracy and thus outperforms VGG-16. This experiment shows that MobileNet is not suitable for in-memory deep learning. Under the same energy budget, MobileNet shows lower accuracy than VGG-16 and ResNet-18. We attribute this to its depthwise layer. When we compute a regular convolution layer, the system reads hundreds of memory cells at once. However, to process the depthwise layer, it only read nine memory cells at once. Therefore, a large portion of the energy is consumed in the peripheral circuits, causing a significant amount of energy overhead.

\setlength{\tabcolsep}{4pt}

\renewcommand{\arraystretch}{1.26}

\begin{table*}[!t]
\caption{Comparison between our solutions and the state-of-the-art on the CIFAR-10 dataset}
\resizebox{1.00\linewidth}{!}{
\begin{tabular}{l  c | c    c  c | c c c   | c c  c}
\hline
\hline
& &\multicolumn{3}{c|}{0\% accuracy drop}&\multicolumn{3}{c|}{1\% accuracy drop} &\multicolumn{3}{c}{2\% accuracy drop}\\
\hline
VGG-16 (93.6\% Acc.)	&Ref.	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	\\
\hline
Binarized Encoding	&\cite{zhu2019configurable}	&378	&74M	&2.8	&135	&74M	&2.8	&94	&74M	&2.8	\\
Weight Scaling	&\cite{ielmini2010resistance}	&444	&15M	&2.8	&78	&15M	&2.8	&49	&15M	&2.8	\\
Fluctuation Compensation	&\cite{wan2020voltage}	&1091	&15M	&14	&157	&15M	&14	&82	&15M	&14	\\
\textbf{Ours (A+B)}	&	&\textbf{36}	&\textbf{15M}	&\textbf{2.8}	&\textbf{16}	&\textbf{15M}	&\textbf{2.8}	&\textbf{11}	&\textbf{15M}	&\textbf{2.8}	\\
\textbf{Ours (A+B+C)}	&	&\textbf{4.1}	&\textbf{15M}	&\textbf{14}	&\textbf{1.0}	&\textbf{15M}	&\textbf{14}	&\textbf{0.5}	&\textbf{15M}	&\textbf{14}	\\
\hline
ResNet-18 (95.2\% Acc.)	&Ref.	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	\\
\hline
Binarized Encoding	&\cite{zhu2019configurable}	&876	&56M	&6.8	&389	&56M	&6.8	&286	&56M	&6.8	\\
Weight Scaling	&\cite{ielmini2010resistance}	&1127	&11M	&6.8	&209	&11M	&6.8	&158	&11M	&6.8	\\
Fluctuation Compensation	&\cite{wan2020voltage}	&2217	&11M	&34	&474	&11M	&34	&347	&11M	&34	\\
\textbf{Ours (A+B)}	&	&\textbf{83}	&\textbf{11M}	&\textbf{6.8}	&\textbf{22}	&\textbf{11M}	&\textbf{6.8}	&\textbf{10}	&\textbf{11M}	&\textbf{6.8}	\\
\textbf{Ours (A+B+C)}	&	&\textbf{6.9}	&\textbf{11M}	&\textbf{34}	&\textbf{1.1}	&\textbf{11M}	&\textbf{34}	&\textbf{0.7}	&\textbf{11M}	&\textbf{34}	\\
\hline
MobileNet (91.3\% Acc.)	&Ref.	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	\\
\hline
Binarized Encoding	&\cite{zhu2019configurable}	&392	&16M	&4.6	&81	&16M	&4.6	&62	&16M	&4.6	\\
Weight Scaling	&\cite{ielmini2010resistance}	&232	&3.2M	&4.6	&57	&3.2M	&4.6	&42	&3.2M	&4.6	\\
Fluctuation Compensation	&\cite{wan2020voltage}	&659	&3.2M	&23	&126	&3.2M	&23	&91	&3.2M	&23	\\
\textbf{Ours (A+B)}	&	&\textbf{75}	&\textbf{3.2M}	&\textbf{4.6}	&\textbf{23}	&\textbf{3.2M}	&\textbf{4.6}	&\textbf{13}	&\textbf{3.2M}	&\textbf{4.6}	\\
\textbf{Ours (A+B+C)}	&	&\textbf{12.2}	&\textbf{3.2M}	&\textbf{23}	&\textbf{1.8}	&\textbf{3.2M}	&\textbf{23}	&\textbf{1.3}	&\textbf{3.2M}	&\textbf{23}	\\
\hline
\hline
\label{t:sota1}
\end{tabular}
}
\end{table*}

\begin{table*}[!t]
\caption{Comparison between our solutions and the state-of-the-art on the ImageNet dataset}
\resizebox{1.00\linewidth}{!}{
\begin{tabular}{l  c | c    c  c | c c c   | c c  c}
\hline
\hline
& &\multicolumn{3}{c|}{0\% accuracy drop}&\multicolumn{3}{c|}{1\% accuracy drop} &\multicolumn{3}{c}{2\% accuracy drop}\\
\hline
ResNet-18(69.8\% Acc.)	&Ref.	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	\\
\hline
Binarized Encoding	&\cite{zhu2019configurable}	&23k (\textcolor{red}{-0.4\%})	&58M	&151	&2338	&58M	&151	&1336	&58M	&151	\\
Weight Scaling	&\cite{ielmini2010resistance}	&21k (\textcolor{red}{-0.3\%})	&12M	&151	&3544	&12M	&151	&1933	&12M	&151	\\
Fluctuation Compensation	&\cite{wan2020voltage}	&71k (\textcolor{red}{-0.3\%})	&12M	&756	&8505	&12M	&756	&4725	&12M	&756	\\
\textbf{Ours (A+B)}	&	&\textbf{1951}	&\textbf{12M}	&\textbf{151}	&\textbf{897}	&\textbf{12M}	&\textbf{151}	&\textbf{659}	&\textbf{12M}	&\textbf{151}	\\
\textbf{Ours (A+B+C)}	&	&\textbf{142}	&\textbf{12M}	&\textbf{756}	&\textbf{71}	&\textbf{12M}	&\textbf{756}	&\textbf{54}	&\textbf{12M}	&\textbf{756}	\\
\hline
ResNet-34 (73.3\% Acc.)	&Ref.	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	&Energy ($\mu$J)	&\#Cells	&Delay ($\mu$S)	\\
\hline
Binarized Encoding	&\cite{zhu2019configurable}	&28k (\textcolor{red}{-0.2\%})	&109M	&207	&2844	&109M	&207	&1778	&109M	&207	\\
Weight Scaling	&\cite{ielmini2010resistance}	&25k (\textcolor{red}{-0.1\%})	&22M	&207	&3302	&22M	&207	&2154	&22M	&207	\\
Fluctuation Compensation	&\cite{wan2020voltage}	&83k (\textcolor{red}{-0.1\%})	&22M	&1033	&7990	&22M	&1033	&4857	&22M	&1033	\\
\textbf{Ours (A+B)}	&	&\textbf{2496}	&\textbf{22M}	&\textbf{207}	&\textbf{1044}	&\textbf{22M}	&\textbf{207}	&\textbf{729}	&\textbf{22M}	&\textbf{207}	\\
\textbf{Ours (A+B+C)}	&	&\textbf{168}	&\textbf{22M}	&\textbf{1033}	&\textbf{90}	&\textbf{22M}	&\textbf{1033}	&\textbf{62}	&\textbf{22M}	&\textbf{1033}	\\
\hline
\hline
\label{t:sota2}
\end{tabular}
}
\end{table*}

\subsection{Robustness to Different Devices}

Today, academia and industry have developed various types of  EMT cells, which have different levels of fluctuation intensity. Hence, it is necessary to prove the robustness of our solutions under any level of fluctuation intensity. In Fig.~\ref{f:rob}, we test our solutions and the state-of-the-art under three intensity levels~\cite{raghavan2013microscopic}:  weak,  normal, and strong. The experiment is conducted on the ImageNet dataset using two ResNet models. All solutions, including the state-of-the-art, are free to tune the energy coefficient $\rho$. We compare the energy consumption when the model achieves its maximum accuracy. Noted that on the ImageNet dataset, our solutions can achieve the same accuracy as the baseline model running on GPU, where the state-of-the-art cannot.

The results show the robustness of our solutions. At any level of fluctuation intensity, our solution has almost the same performance on energy reduction. When fluctuation intensity is increased, both ours and the state-of-the-art solutions prefer a higher energy coefficient $\rho$ to maximize the model accuracy, resulting in a higher energy consumption. However, our solutions still outperform the state-of-the-art. Our solutions A+B and A+B+C shows one and two orders of magnitude energy reduction, respectively. From this point onwards, we shall assume the fluctuation intensity to be normal. Results under any other intensity level follow a similar trend.

\subsection{Verification of the Optimization Solutions}

In Fig.~\ref{f:sota}, we verified our optimization solution by testing two ResNet models on the ImageNet dataset. The dashed line in the figure shows the baseline accuracy. We defined it as the highest accuracy we can achieve on GPU. We also list the accuracy of the state-of-the-art for comparison. Among all, our solution A+B+C has the highest top-1 and top-5 accuracies, which are the same as the baseline accuracies. The accuracy of solution A+B also shows higher accuracy than the state-of-the-art. We can observe a small accuracy loss under a smaller energy budget. By contrast, models optimized by the state-of-the-art have significant accuracy losses. For example, we can see at least 0.9\% and 0.8\% top-1 accuracy losses of the ResNet-18 and ResNet-34, respectively. We can also see that the ResNet-18 on the ImageNet dataset consumes more energy than the same model on the CIFAR-10 dataset. It is because ImageNet has a larger image size than CIFAR-10.

\subsection{Holistic Comparison with the State-of-the-Art}

Our proposed solutions have better performance not only on energy reduction but also reduce cost and latencies. In Table~\ref{t:sota1} and Table~\ref{t:sota2}, we give a holistic comparison of our solutions with the state-of-the-art on energy consumption, the number of cells, and latency, under the same accuracy loss. We test various models on the CIFAR-10 and the ImageNet datasets. From the tables, our method has the lowest energy consumption, the least number of cells, and the shorted latency. Specifically, solution A+B shows one order of magnitude improvement in energy consumption, and solution A+B+C can achieve two orders of magnitude improvement, compared with the state-of-the-art. However, one limitation of Solution A+B+C is that its latency is longer because the low-fluctuation decomposed computation takes time to accumulate the results. The trade-off between energy consumption and latency depends on the specific application. Therefore, we list the results of solution A+B and solution A+B+C. Developers can choose one of them based on their demands.

Another observation is that our solutions are the only ones that can fully recover the accuracy loss on the ImageNet datasets. As can be seen from the third column of Table~\ref{t:sota2}, all the state-of-the-art solutions are unable attain no accuracy loss (we mark their actual accuracy drops in red text). In a neural network, some parameters are vital to achieving high model accuracy. However, the state-of-the-art usually applies a general optimization rule to all the model parameters. After the optimization, those important parameters still have relatively large fluctuations, which constraints the recovery of model accuracy. Unlike the state-of-the-art, our solutions can automatically identify those significant parameters and minimize their fluctuations.

\section{Conclusion}
\label{s6}

In-memory deep learning has a promising future in the AI industry because of its high energy efficiency over traditional deep learning. This is even more so if the potential of emerging memory technology (EMT) especially in analog computing mode is used. Unfortunately, one of the major limitations of EMT is the intrinsic instability of EMT cells, which can cause a significant loss in accuracy. On the other hand, falling back on a digital mode of operation will erode the potential gains. In this work, we propose three optimization techniques that can fully recover the accuracy of EMT-based analog in-memory deep learning models while minimizing their energy consumption: They include the device-enhanced dataset, energy regularization, and low-fluctuation decomposition. Based on the experiment results, we offer two solutions. Developers can either apply the first two optimization techniques or apply all three to the target model. Both solutions can achieve higher accuracy than their state-of-the-art counterparts. The first solution shows at least one order of magnitude improvement in energy efficiency, with the least hardware cost and latency. The second solution further improves energy efficiency by another order of magnitude at the cost of higher latency.

redpajama_set_name: RedPajamaArXiv


text:
Q: How can I set the states (names) after using markovchainFit for plotting? here is my contrived dataset/logic to plot markovchain
sequences <- list(
      c("Opened", "2709342", "END"),
      c("Opened", "3067630", "END")
    )

    sequencesMarkovchainFit <- function(sequences){
      # Create a continuous sequence string with each sequence having START and END
      #
      # sequences <- list( c("A"),
      #                    c("B"))
      # sequence = c("START", "A", "END", "START", "B", "END" ...)
      sequence <- c ()
      for (i in 1:length(sequences)){
        #sequence <- c(sequence, "START", unlist(sequences[i]), "END")
        sequence <- c(sequence, "START", unlist(sequences[i]))
      }
      fit <- markovchainFit(data=sequence)
      chain <- as(fit$estimate, "markovchain")
      return(chain)
    }

    sequencesMC <- sequencesMarkovchainFit(sequences)

    plot(sequencesMC, main="Selected Sequences T (transition matrix)")

and the resulting plot still has the indices (i.e. 0,1,2..) instead of the desired states (names)
plot of markovchain after creating with markovchainFit


text:
Truck Talk: Electrical storm edition
This week, we chase storm clouds gathering for electric pickup trucks not wearing a Ford blue oval, speculate on who a big customer at Plus might be and review how Daimler Truck is setting the stage for its independence.
Storm clouds for electric pickups
Now that Ford Motor Co.(NYSE: F) has revealed the F150 Lightning electric truck, it is — or should be — panic time for startups eyeing the battery-powered commercial pickup segment.
At the top of the list, with arguably the most to lose, is Lordstown Motors Corp. (NASDAQ: RIDE). The de-SPACed startup plans to start building Class 1 electric pickups at a former General Motors plant late this fall.
Lordstown (LMC) already has attracted Securities and Exchange Commission scrutiny and daggers from short seller Hindenburg Research, which deconstructed the company's claims of 100,000 preorders and alleged other shady dealings.
But its biggest threat to LMC's viability comes from Ford, which has sold more pickup trucks than any company on the planet for the past 44 years. For 40 of those years, it was the best-selling vehicle of any kind.
The F-150 is the profit engine for the Dearborn, Michigan-based automaker. Like college football helps pay for other less-popular sports, F-Series profits help cover the cost of the rest of the lineup. Given Ford's size, scale and know-how, it is practically unbeatable in the pickup space.
"I think that there's little or no chance that Lordstown is going to have any market share of any significance," Guidehouse Insights principal analyst Sam Abuelsamid told Truck Talk. "The F-150 has a $12,000 cheaper base price; it has a service network; it's compatible with all the stuff that Ford truck owners already use."
Dire assessment
A more dire assessment comes from Wolfe Research analyst Rod Lache, who downgraded LMC stock to "underperform" following the Lightning reveal. He set a price target of $1 a share.
From a PR perspective, LMC shares got a boost when the company announced a week of factory tours, presentations and test drives of the Endurance pickup in June. It is billed as the world's first all-electric commercial pickup. But will it live up to its name?
Former Vice President Mike Pence at the Lordstown Endurance reveal in June 2020. (Photo: Lordstown Motors)
The Lightning reveal on Wednesday evening is backed by a national TV campaign pointing to its spring 2022 launch. That marketing muscle and Wolfe's dour projection led LMC shares 15.28% lower Thursday to $9.70. In premarket trading Friday, they were down another 5.14% to $9.14.
Calling out LMC is obvious since it markets itself as a commercial entry. Where does the Lightning leave other electric pickups like the Tesla Cybertruck and the Rivian R1T?
History may help answer that. The Detroit 3 have long dominated the full-size pickup truck segment. Even as Japanese and other Asian automakers wrested passenger car leadership away from the former Big Three in the1980s and '90s, they have yet to make serious inroads in full-size pickups.
Detroit manufacturers Ford, Ram and GM sold 2.29 million full-size pickups in 2020. Toyota, Honda and Nissan sold a combined 174,000.
One bit of irony: GM invested both cash and property (its former Lordstown Assembly complex) in LMC. GM's own full-size electric Chevrolet Silverado is coming in 2023 or 2024.
Who's that customer?
Autonomous trucking software developer Plus finally got its long-rumored SPAC deal with Hennessy Capital on May 10, setting it up to be the second startup to go public after TuSimple's successful initial public offering in April. Deep inside the Plus investor deck (page 35 to be exact) is a vague description of what might or might not be a big deal.
An unnamed customer that evaluated multiple autonomous driving software developers chose Plus as its provider of autonomy-enabled trucks in 2019. It signed a master purchase agreement (MPA) in January with the promise of getting at least some of 1,000 PlusDrive units between February of this year and 2023.
The MPA includes a term sheet that requires the mystery customer to purchase 10,000 PlusDrive units in exchange for a fully vested stock warrant for up to 20% of Plus shares. Few fleets come close to needing so many retrofits. One name comes to mind.
Amazon Smile?
Amazon (NASDAQ: AMZN) recently agreed to purchase renewable natural gas (RNG) from Clean Energy Fuels (NASDAQ: CLNE). As part of the deal, it got a warrant to purchase up to 19.99% of Clean Energy shares.
The fine print of the Plus deal contains all of the expected language about how it might not happen and mentions that no definitive warrant agreement has been entered yet.
Is Amazon the mystery customer for Plus? (Photo: Jim Allen/FreightWaves)
The fracking truth
Startup hybrid driveline maker Hyliion Holdings (NYSE:HYLN ) is partnering with frac sand hauler Detmar Logistics LLC to start the the oil and gas industry servicer on a five-year path to electrifying its fleet. Detmar ordered 10 Hyliion Hybrid Electric units for retrofit on its Class 8 Volvo models. More orders may follow if the first conversions work out.
Hyliion, which touts its coming ERX Hypertruck as capable of achieving negative net-zero carbon emissions, thinks it can help Detmar clean up the part of the fossil fuels industry that requires ground transportation.
Flaring at oil and gas extraction sites is one of the largest greenhouse gas-emitting practices in the industry. Capped and captured, the emissions can be made into the negative net-zero renewable natural gas fuel that Hyliion wants to use.
(Photo: Jim Allen/FreightWaves)
Virtual roadshow
Daimler Truck gave some strong hints this week of what it will look like as a stand-alone company following its spinoff from Daimler AG and listing on the Frankfurt Stock Exchange later this year.
Among the business goals described in a virtual roadshow-style event on Thursday, the new company is targeting double-digit operating margins by 2025 to close its profitability gap with rivals Volvo AB (OTC: VLVLY) and PACCAR Inc. (NASDAQ: PCAR).
Daimler will do what it takes to target benchmark performance in every region, according to CEO Martin Daum, who recently signed a contract extension to lead the new entity. That includes Daimler Trucks North America, which provides much of the current Daimler Truck AG's profits through the market dominance of the Freightliner brand.
In a presentation, Daimler Truck described plans to reduce fixed costs, capital expenditures and research development spending 15% by 2025 compared to 2019. That includes $366 million in personnel costs reductions by 2022. Look for a greater focus on more profitable heavy-duty trucks. One sign of that is the recent outsourcing of medium-duty engines to Cummins Inc. (NYSE: CMI).
Daimler has a fuel cell manufacturing joint venture with rival Volvo that the Swedish truck maker paid $700 million for a 50% stake. That reduces Daimler Trucks' outlay for hydrogen-powered vehicles that both it and Volvo plan to produce in the second half of the decade.
Autonomous trucking software startup TuSimple threw a little shade on partner Navistar during its first-quarter earnings call, saying its order for 25 new trucks expected in March won't arrive until June or July. Truck Talk asked Navistar if it was experiencing other delays in delivering trucks because of a well-chronicled microchip shortage and other supply chain disruptions.
"TuSimple placed its order in January, a time at which the order board was already seeing significant lead time," a Navistar spokeswoman said. "The delay has been amplified by TuSimple's request for major modifications to the order."
Want to to get Truck Talk delivered weekly on Fridays? Subscribe here and you'll be added to the distribution (except for a vacation break next week.) Truck Talk will return on Friday, June 4.
That's all for now. See you in two weeks.
Newer Freight brokers urged to increase security in light of pipeline cyberattack
Older Kansas City Southern chooses CN bid, terminates CP merger agreement


text:
Francesc Vergés i Vives (Sant Feliu de Llobregat, 9 de març de 1919 - Barcelona, 16 de febrer de 2015) va ser un sacerdot catòlic, antifranquista i catalanista, vinculat al monestir benedictí de Sant Pere de les Puel·les.

Biografia
Va néixer al carrer Torres i Bages de Sant Feliu de Llobregat i va estudiar els primers anys al Col·legi Model. Entrà al Seminari de Barcelona, s'ordenà sacerdot i marxà a Roma. Al seminari i a Roma va fer amistat amb l'eclesiàstic Manuel Bonet i Muixí, fundador de la Unió Sacerdotal, amb qui va assistir, el 1933, al funeral del president Francesc Macià. Va substituir mossèn Pere Tarrés a Sant Esteve de Sesrovires i coincidí amb mossèn Joan Batlles a Vilafranca del Penedès, amb qui participà activament en la recuperació de les colònies per als infants juntament amb capellans de la Federació de Joves Cristians de Catalunya.

Defensor de la democràcia i participant en actes com la Caputxinada, era mal vist pel Franquisme, que a principis de la dècada del 1960 el segrestà al carrer Anglí i el portà a la comissaria de la Verneda. Formava part de la llista negra dels 462 catalans més sospitosos pel Franquisme, segons la qual era progressista i catalano-separatista.

Des del 1947 fins que una embòlia li ho va impedir, va ser el capellà de la comunitat benedictina de Sant Pere de les Puel·les. Visqué a la casa sacerdotal del monestir, impulsà la formació prematrimonial, la dels seminaristes, el Tribunal Eclesiàstic de Barcelona i la Unió Sacerdotal.

Referències

Preveres barcelonins
Antifranquistes catalans
Santfeliuencs
Religiosos barcelonins contemporanis
Morts a Barcelona
Activistes barcelonins
Activistes catalans del sud contemporanis

redpajama_set_name: RedPajamaWikipedia


text:
Attorney investigated after house fire
by KEITH KINNAIRD
Hagadone News Network | April 16, 2010 9:00 PM
Copyright: Bonner County Daily Bee
VAY - A Bonner County attorney awaiting trial in a federal tax fraud case is being investigated for attempted murder, arson and molestation after his home was destroyed by a fire last month.
David Charles Jacquot was arrested on a federal warrant on March 16 for violating the terms of his conditional release in the tax fraud case, according to U.S. District Court records.
Jacquot's arrest followed a March 11 fire at his Bandy Road home. The investigation uncovered a cache of firearms, including a 40-millimeter grenade launcher, the federal warrant said.
Bonner County Prosecutor Louis Marshall said Jacquot is suspected of molesting his adopted daughter from Kazakhstan.
"Mr. Jacquot is a suspect in the child molestation," said Marshall.
Marshall declined to confirm if Jacquot is suspected of the other crimes.
However, federal court documents indicate the Bonner County Sheriff's Office is investigating Jacquot for arson, attempted murder and insurance fraud in connection with the early morning structure fire.
Jacquot, 47, was held briefly at the Bonner County Jail and then transferred to federal custody, Marshall said.
The U.S. Department of Justice's Federal Bureau of Prisons Web site listed Jacquot as an inmate at a facility in SeaTac, Wash., on Tuesday. He was not listed as an inmate at any federal institution on Thursday.
Jacquot is due in U.S. District Court in San Diego on May 10 for a status conference in his tax fraud case, records show.
It was unclear if Jacquot has legal counsel representing him in the Idaho matters. Jacquot's defense counsel in the tax case, Michael Crowley of San Diego, could not be reached after hours on Thursday. E-mail and phone messages were not immediately returned.
The adopted daughter, who is in her late teens, disclosed the molestation to authorities on March 3, said Gordon Hodge, who is the cousin of David Jacquot's wife, Ann.
Hodge said the teen told him David Jacquot had been molesting her since the age of 13, shortly after she arrived in the country.
"I had suspected that for years, but never had gotten any proof," said Hodge, who contacted Bonner County sheriff's officials about the teen's disclosures to him on March 8.
Three days later, Hodge said he received a frantic phone call from his cousin, who said she escaped the burning home, but that David Jacquot could not be found and feared he was dead. The couple's biological son, Nate, had also escaped the blaze.
Hodge said the couple's adopted daughter had been staying at the home of friends when the fire broke out.
Hodge, who ended up going to the scene of the fire, said a firefighter later encountered David Jacquot, who had a hand injury and seemed unable to speak.
"They found him wandering around, dazed and confused," said Hodge.
Bonner Dispatch records indicate the fire was called in at 1:17 a.m.
When the sun rose, Hodge said he discovered an array of peculiar items strewn about near an outbuilding directly across the driveway from the home and took photos of them.
The photographs show a flashlight on a utility trailer, a walkie talkie, a broken pair of eyeglasses Hodge said belong to David Jacquot, blood smears on doors leading to the shed and a small trail of blood. There were also spent .22-caliber casings and live 9-mm rounds.
The photos also show the barrel and slide of a Glock, in addition to the semiautomatic pistol's loaded magazine. Hodge said a fire truck had apparently run over the weapon during the chaos from the fire.
Hodge said he also discovered the circuit breaker to the well had been shut off. He doubts the fire tripped the breaker because it's separate from the home's electrical system and the switch was fully engaged in the off position instead of a tripped position. A usually open dog door had also been secured, he said.
Hodge, who used to live in another dwelling on the property, said David Jacquot kept a vast array of weaponry and ammunition in both homes. Hodge said the homes were connected by an underground passage.
A condition of his release in the tax case forbid Jacquot from possessing firearms, although that condition was later modified, according to federal court records.
"He had petitioned the judge in his case to let him have a gun to protect livestock from predators. And instead of having a gun, he had an arsenal of guns. He had so many bullets the police didn't count them - they weighed 'em," said Hodge.
Documents in the federal case indicate David Jacquot was found to be in possession of a Glock 9-mm with an obliterated serial number and more than 40 other weapons, including the grenade launcher.


text:
Kelbadjar ou Kalbajar () est une ville située dans le raion de Kelbadjar en Azerbaïdjan. De 1993 à 2020, elle portait le nom de Karvachar ou Qarvachar (en ) et était la capitale de la région de Chahoumian au Haut-Karabagh.

Le 25 novembre 2020, conformément à l'accord de cessez-le-feu qui a mis fin à la seconde guerre du Haut-Karabagh, la ville, ainsi que le district environnant, ont été restitués à l'Azerbaïdjan.

Géographie
Située dans la vallée du Tartar, elle se trouve à  de la capitale Bakou.

Démographie
Elle comptait  avant sa prise par les forces arméniennes le 2 avril 1993, pendant la première guerre du Haut-Karabakh, qui fut suivie d'importants mouvements de population. En 2010, elle comptait 600 habitants.

Histoire

Conflit entre l'Arménie et l'Azerbaïdjan
La ville est occupée par les forces arméniennes à la suite de leur victoire dans la première guerre du Haut-Karabagh, qui prend fin en 1994. La ville ne fait pas partie de l'ancien oblast autonome du Haut-karabagh, mais se trouve sur le corridor reliant celui-ci à la république d'Arménie.

La résolution n° 822 du Conseil de sécurité de l'ONU exige en 1993 le retrait de ce territoire des forces armées locales arméniennes. Un accord proposé par le groupe de Minsk de l'OSCE (« accord global », ), accepté par les gouvernements arménien et azéri, prévoit le retrait des troupes aux frontières de 1988, c'est-à-dire, à celles de l'oblast autonome du Haut-Karabagh et de la république socialiste soviétique d'Arménie pour les Arméniens et aux frontières de la république socialiste soviétique d'Azerbaïdjan pour les Azéris, sans compter l'oblast, en attendant l'aboutissement des négociations.

A la fin de la seconde guerre du Haut-Karabagh, et dans le cadre de l'accord signé entre l'Azerbaïdjan et l'Arménie le 10 novembre 2020, Kelbajar doit être restituée par l'Arménie à l'Azerbaïdjan le 15 novembre suivant. Des habitants arméniens expropriés incendient leur maison plutôt que de les abandonner aux Azerbaïdjanais.

Selon la déclaration trilatérale signée par les présidents de la République d'Azerbaïdjan, de la Fédération de Russie et le Premier ministre de la République d'Arménie, les unités de l'armée azerbaïdjanaise sont entrées dans la région de Kelbajar le 25 novembre.

Notes et références

Ville en Azerbaïdjan


Usagi Circus Show Time (ウサギサーカスShow Time), Rabbit Circus Show Time is a song featured in "Yume Senshi – Ai – Eien ni…" and its revised edition.
The world's finest Rabbit Circus!
How are you finding it? The entertainment?
It's the world's finest Crazy Show Time!
It's the world's finest Rabbit Circus!
How's the impression on you?
It's the long-awaited world's finest Crazy Show Time!
At last, they've arrived! The Star Couple!


text:
Q: Why is Airplay from my iPhone to my Apple TV so slow? I got the new Apple TV and noticed that it takes ages to stream a video from my iPhone.
Moreover, streaming videos directly from youtube on my iphone over airplay may cause the video to be very blurry for a few secs and then back to normal (HD), this might be due to buffering but im not sure if the internet connection has an impact on that.
Is there a way to improve the efficiency and speed of the device?
I am using a Wireless N router. The Apple TV is connected via WIFI. The iPhone is connected wireless through airplay.

A: The crucial issue is the speed of your router. Ideally you'd have the Apple TV connected to a router via an Ethernet cable using Gigabit Ethernet (1000/baseT) and your router would have WiFi N, not Wifi G (which is slower by a factor of about five), for the iPhone connection. Then you'd want to make sure that the WiFi N router was operating on a channel not subject to interference from other routers nearby, or from physical interference that would lead to signal attenuation.
It would be a good idea to try putting your WiFi router in the same room as your Apple TV and your iPhone and to see if video image quality improves. If so, this is an indication that you were experiencing WiFi signal attenuation due to interference caused by the materials used in the walls of the building.
Also, if this is the first time you've used an AppleTV, and the first time you've tried to watch HD video streaming over the Internet on your television, it may be that your Internet connection speed from your Internet service provider simply isn't fast enough to support HD video. If you have DSL, you may wish to consider upgrading to a cable modem or fiber-optic service with higher bandwidth. This year I found that AT&T had made their Uverse fiber-optic service available in my neighborhood. I upgraded to Uverse and got Internet service at four times the speed of my old AT&T DSL service for the same price I was paying for the slower DSL.

A: This worked. After 3 days of going over the same posts and no clear answer, The problem is in the appleTV software. revert to version 5.1.1 and it works great. I'm tired of people posting the same answers like we don't know any better to check the wireless router etc... any ways. reset the Apple TV and make sure it does not update to new software.
My apple TV took upto 2 min to see the Ipad screen.


Using only the best local hardwoods found in South Florida. We hand craft & carve each base. First, we chainsaw, hand sand, polish and apply several coats of polyurethane to create a stunning standing mount or tournament trophy.
Please Note : All wood bases vary in both size in shape & grain of wood. No two, are the same.
Each Sculpture & Base packaged separately, and fit together by firmly pushing down into custom made slots in wood.
Both Sculpture and Base are marked for easy mounting.