The Global Carbon Capture and Storage Institute Ltd (Global CCS Institute) (ABN 136 614 465) is subject to the Privacy Act 1988 (Cth), which requires that we comply with the Australian Privacy Principles (APPs) in dealing with your personal information.
The Global CCS Institute only collects such information as it requires to perform our functions (set out in the Global CCS Institute's constitution as amended from time to time in accordance with the Australian Corporations Act).
The Global CCS Institute only uses and discloses personal information for the purposes of performing our functions, conducting our business, communicating with individuals and complying with our legal obligations. This may include the dissemination of information to individuals, and facilitating and administering projects.
The Global CCS Institute will not disclose personal information to third parties unless the consent of the individual concerned is obtained, the disclosure is required by law, or the disclosure is otherwise permitted under the Privacy Act.
No attempt will be made to identify individual users or their browsing activities except in the event that it is reasonably necessary for the enforcement of criminal law. In the case of the latter, we will comply with the Privacy Act 1988 (Cth), regarding the limits and protocols associated with disclosure of personal information. The Institute may collect your personal information when you transmit information to the Institute through the website (e.g. if you become a registered user) or by email.
If you are a registered user, your details and comments that you post on the website will be available on publicly accessible areas of the site. The Global CCS Institute will not sell, rent or otherwise disclose your user details to third parties in any manner not set out in this Privacy Statement. While the Global CCS Institute takes all reasonable measures to ensure information transmitted over its website is secure, users should be aware that there is an inherent risk that transmitted information could be intercepted by third parties.
No attempt will be made to identify anonymous users or their browsing activities unless the Global CSS Institute is legally compelled to do so.
The Global CCS Institute is a global organisation. The authorised use and disclosure of personal information may include disclosure to overseas recipients. The countries in which such recipients are likely to be located include Belgium, Canada, China, Japan, and the United States of America.
You can request access to your own personal information held by the Global CCS Institute. There is no charge for making such a request, but the Global CCS Institute may charge a reasonable fee for the work involved in locating, copying and providing access to the information. Reasonable requests for amendment or annotation of your personal information records will be considered and accommodated as far as possible.
If you have concerns about the privacy of your personal information, you should contact the Global CCS Institute in the first instance on+61 3 8620 7359. If this does not produce a satisfactory outcome, you can contact the Office of the Australian Information Commissioner by telephone on 1300 363 992, or email enquiries@oaic.gov.au.


text:
woodrow wilson news, opinion and analysis - Macleans.ca
How 1916 set the stage for America to enter WWI
The U.S. greeted the outbreak of the First World War with disbelief, but by late 1916 thousands of Americans were already personally involved in the war
By A. Scott Berg
Jim Flaherty: 'Do it for your country'
The prepared text of a speech—an interesting, perhaps even charming, and apparently quite personal speech—delivered today by Finance Minister Jim Flaherty to students at the University of Western Ontario's Ivey
Where have you gone, Rutherford B. Hayes?
The President gets to play rock star once a year with the Vice-President and the Speaker of the House as his rhythm section
Was Obama's Nobel for "awesomeness" and positive thinking?
The President's win is like 'The Secret' being unleashed on the worldwide political stage

redpajama_set_name: RedPajamaCommonCrawl


text:
package freeblogdaemon

import (
	"os"
	"log"
	"io/ioutil"
	"path/filepath"
	"github.com/flosch/pongo2"
)

type Blog struct {
	Title string
	Subtitle string
	BlogPath string
	BlogResultPath string
	PostTemplatePath string
	AboutTemplatePath string
	MainTemplatePath string
}

func (blog Blog) CreateAboutContext(aboutPage Post) pongo2.Context {
	return pongo2.Context{"page": "about", "title": blog.Title, "subtitle": blog.Subtitle, "aboutPage": aboutPage}
}

func (blog Blog) CreatePostContext(post Post) pongo2.Context {
	return pongo2.Context{"page": "post", "title": blog.Title, "subtitle": blog.Subtitle, "post": post}
}

func (blog Blog) CreateMainContext(posts []Post) pongo2.Context {
	return pongo2.Context{"page": "main", "title": blog.Title, "subtitle": blog.Subtitle, "posts": posts}
}

func WriteFile(filePath string, stringToWrite string) error {
	err := ioutil.WriteFile(filePath, []byte(stringToWrite), 0644)
	return err
}

func ReadAboutAndCreateBlog(aboutPath string) (Post, error) {
	aboutPage, err := ReadBlogPost(aboutPath)
	if err != nil {
		return Post{}, err
	}
	aboutPage.Path = "/about.html"

	return aboutPage, nil
}

func CreateBlogObj(config BlogConfig) Blog {
	blog := Blog{}
	blog.Title = config.Title
	blog.BlogPath = config.Path
	blog.Subtitle = config.Subtitle
	blog.BlogResultPath = config.ResultPath
	blog.PostTemplatePath = filepath.Join(config.TemplatePath, "/post.tpl")
	blog.AboutTemplatePath = filepath.Join(config.TemplatePath, "/about.tpl")
	blog.MainTemplatePath = filepath.Join(config.TemplatePath, "/main.tpl")

	_, err := os.Stat(blog.AboutTemplatePath)
	noAbout := os.IsNotExist(err)

	_, err = os.Stat(blog.PostTemplatePath)
	noPost :=  os.IsNotExist(err)

	_, err = os.Stat(blog.MainTemplatePath)
	noMain :=  os.IsNotExist(err)

	if noAbout || noMain || noPost {
		log.Fatal("Not all templates present in " + config.TemplatePath)
	}
	return blog
}

func WriteAboutPage(blog Blog, aboutPath string) error {
	aboutPage, err := ReadAboutAndCreateBlog(aboutPath)
	if err != nil {
		return err
	}

	aboutContext := blog.CreateAboutContext(aboutPage)
	tmplOut := pongo2.RenderTemplateFile(blog.AboutTemplatePath, aboutContext)

	err = WriteFile(filepath.Join(blog.BlogResultPath, "about.html"), tmplOut)
	return err
}

func WritePosts(blog Blog, maxNewPostsSize int, postsFolderPath string) ([]Post, error) {
	resultPostsFolderPath := filepath.Join(blog.BlogResultPath, "/posts/")
	err := os.MkdirAll(resultPostsFolderPath, 0755)
	if err != nil {
		return []Post{}, err
	}

	postsPath := filepath.Join(postsFolderPath, "/*.markdown")

	matches, err := filepath.Glob(postsPath)

	if err != nil {
		return []Post{}, err
	}
	var newestPosts []Post
	for i := len(matches)-1; i >= 0; i-- {
		match := matches[i]

		blogPost, err := WritePost(match, blog)

		if err != nil {
			log.Println(err)
			continue
		}

		if len(newestPosts) == maxNewPostsSize {
			newestPosts = newestPosts[:len(newestPosts)-1]
		}
		newestPosts = append(newestPosts, blogPost)
	}
	return newestPosts, nil
}

func WriteMain(blog Blog, newestPosts []Post) error {
	mainContext := blog.CreateMainContext(newestPosts)
	mainOut := pongo2.RenderTemplateFile(blog.MainTemplatePath, mainContext)

	mainPath := filepath.Join(blog.BlogResultPath, "/index.html")
	err := WriteFile(mainPath, mainOut)
	if err != nil {
		return err
	}

	return nil
}

func WritePost(postPath string, blog Blog) (Post, error) {
	blogPost, err := ReadBlogPost(postPath)
	if err != nil {
		return Post{}, err
	}

	blogPost.Path = filepath.Join("posts", blogPost.Path)

	postContext := blog.CreatePostContext(blogPost)

	postOut := pongo2.RenderTemplateFile(blog.PostTemplatePath, postContext)

	err = WriteFile(filepath.Join(blog.BlogResultPath, blogPost.Path), postOut)
	if err != nil {
		return blogPost, err
	}

	return blogPost, nil
}

func CopyStaticsToResults(templatesPath string, resultPath string) error {
	err := os.MkdirAll(resultPath, 0755)
	if err != nil {
		return err
	}

	var WalkFunc = func(path string, fInfo os.FileInfo, _ error) error {
		return CopyToFolder(path, templatesPath, resultPath)
	}

	err = filepath.Walk(templatesPath, WalkFunc)
	if err != nil {
		return err
	}

	return nil
}

redpajama_set_name: RedPajamaGithub


Copywriting matters to SEO — you know it, I know it, the entire SEO world knows it.
But when it comes to conversions, copywriting is equal parts art and science. There's no foolproof way to write to encourage conversions, because a lot of methods are specific to certain industries, or even certain companies. So how do you pin down what will work for you?
You already know the essentials of writing well — these five copywriting strategies will supplement what you know so you can encourage site visitors to convert.
Fact: Headlines get clicks. When you promote your content through channels like social media, your headline is what gets someone's attention. That means it's up to your headline to decide whether people read what you have to say or skip it completely.
That brief, fleeting moment has to convince someone to take time out of their day and read what you have to say. On the Internet, where new content is produced at break-neck speeds, just standing out against the background noise is hard enough, much less drawing a click.
Still, that's hard. And some ideas can be really good, but only hit two or three of those tenants. In fact, you probably won't hit all four qualities in the same headline — sometimes, you might only get two or three. But you can still have a really good idea.
This headline is unique (never seen it before), ultra-specific (eight actors), and urgent (the movies are coming out soon). However, the usefulness of the headline is debatable. If you're a cinemaphile, this headline might be the most useful thing you read all day, and you might even make time for it. Outside of that niche, it's probably just an interesting read that you'll click if it catches your eye.
However, if we just take that effort one step further to hit all four U's, headlines become incredibly effective.
The above headline works because it's unique (who's seen that before?), ultra-specific (six big-name companies), urgent (they're dying — not yet dead), and useful (you might be a customer!).
If you need proof of its effectiveness, it has the view count right above the headline. Cracked published that article on June 17, and we took the screenshot on June 23. 1.8 million readers in less than a week? Pretty impressive.
And admit it — as a savvy consumer, you want to know if your favorite company is on that list, right?
The "So What?" test is one of the easiest copywriting techniques for writers in any industry. Whenever you write something, ask "so what?" at some point in its production. Question your work's validity and why it matters. If you can answer the question clearly in one sentence, you're good to go — plus, you can keep readers engaged throughout your text so they don't click, start reading, and leave.
If your answer falls short of your standards, then it's back to the drawing board.
This test is important because you need to clearly see the value of your work if you want your readers to do the same. Clearly stating your point and following it up with supporting evidence makes it easy for your readers to keep reading and follow your train of thought.
For example, the point of this blog post is to show readers — that's you — copywriting techniques that can help increase conversions. When you finished the third paragraph, you probably had a good idea about what the rest of the article would say.
After you've determined the value of what you're saying, you can move on to the meat and potatoes of conversion.
As you're writing, you naturally want to keep your readers on the page — but you also want to convince them to convert. Doing both at the same time can be tricky without the right know-how.
Reciprocity – If someone feels like they owe you, they'll pay you back.
Scarcity – If your product is rare or unique, customers will want it.
Authority – If you're experienced, customers will trust you.
Consistency – If you start someone off small and work them up to the main product, they'll buy the main product.
Consensus – If people think others are behaving a certain way, they're more likely to behave the same.
Liking – If people like you, they'll buy from you.
A lot of businesses incorporate these ideas into their websites, but they don't point them out in their copywriting. Offering a free gift (reciprocity), proving you're unique (scarcity), stating your experience (authority), offering a trial download (consistency), showing the number of converted customers (consensus), and connecting with readers (liking) can all boost your conversion rates.
Basically, you have to pull out all the stops when you're talking to potential customers. If you want, you can use all six of these qualities together, plus a call to action to wrap everything up. Let's say we're writing a landing page for software that handles American taxes.
That handful of sentences may not be world-class copywriting, but it can still attract site visitors and persuade them to convert — especially if you add another idea into the mix.
Transparency may sound like it's more of a business decision than a copywriting style, but the decision itself is reflected in what and how you write. There are some huge advantages to transparency, the biggest of which is the trust you establish with your readers. After all, what's the harm in a little disclosure?
Your level of transparency is ultimately up to you. However, one company — Buffer — credits transparency with their sharp and intense rise in profitability. The company has only been around for four years, yet it has 2,000,000 registered users for its software. Buffer's unusual openness extends to everything about their business, including how they operate, how much each employee makes, and even how often they sleep.
Sure, lots of other factors play into Buffer's success as well. But the fact that they're so open about their company establishes trust and a certain level of value within the community around them. If they're publically sharing their employees' salaries (and talking about it), what could they possibly have to hide?
Transparency works so well in part because it makes customers feel like you're leveling with them. No smoke, no mirrors, no tricks — just your down-to-earth business. When they know that you've told them everything about your business's operations, you effectively remove any reservations they could have about converting. And when you take away the reasons someone might not convert, you make it way easier for them to follow through.
The effects of positivity on branding are well-documented, and maintaining a positive attitude online can help you reach and interact with more people. In fact, an overall positive sentiment on Twitter can help you get more followers, while the opposite — broadcasting negative sentiments too often — can actually make people unfollow you.
The same idea carries over into copywriting. Words have power, particularly power over moods, and positive wording can foster a better brand association for your business.
Positive language tends to be more personal, direct, solutions-oriented, happy, and easy to read than negative speech. In fact, positivity is so effective that it actually impacts consumer loyalty more than trust, and that's purely based on how you make a customer feel.
The feelings of your customers matter, and by appealing to their optimism, you can encourage them to convert from a place of trust, or at least preference. Engaging customers optimistically comes with the added bonus of appealing to them emotionally, which, again, urges them to convert based on how they feel.
By utilizing these five copywriting strategies, you can help boost your conversions to meet long-term company goals. Sticking to these five ideas can give your blogs, landing pages, articles, newsletters, and more the extra push they need to help your bottom line while connecting with your customers.
Have you tried any copywriting techniques to boost your conversions? Let me know in the comments!
And you get bonus points if you can single out where I used each of these techniques in the post.


text:
\section{Introduction}

In the last five years there have been several efforts to integrate neural networks (NNs) in the solution of the incompressible Navier-Stokes equations following different approaches. For turbulent flows, the most common approach is to derive data-driven turbulence closure models. For example, \citet{ling2016reynolds} proposed a data-driven Reynolds-averaged Navier–Stokes (RANS) turbulence closure model by embedding Galilean invariance into deep neural networks and demostrated better accuracy for  predicting the Reynolds stresses. Similarly, \citet{wang2017physics} used random forest regression to predict the discrepancies of the baseline RANS-predicted Reynolds stresses compared to those from the DNS data, hence predicting the  Reynolds stresses with high accuracy. \citet{jiang2020novel} developed a novel RANS stress closure with machine-learning-assisted parameterization and nonlocal effects, aiming at reducing both structural and parameteric inaccuracies and achieving a more appropriate description for Reynolds stress anistropy. For large-eddy simulation (LES) of isotropic turbulence, \citet{zhou2019subgrid} developed a data-driven subgrid scale model by using NNs with only one hidden layer. In addition, some reduced order models (ROMs) or fast prediction models in fluid mechanics have also been investigated. For example, convolutional neural networks (CNNs) were used to construct the prediction model of cylinder wake  in \citep{jin2018prediction}, and a temporal CNN was used to establish a data-driven model for predicting the coefficients of proper orthogonal decomposition (POD) modes of cylinder wake in \citep{wu2020data}. The bidirectional recurrent neural networks were employed to predict the POD coefficients of cylinder wake based on a few velocity measurements \citep{jin2020time}, obtaining more accurate results than the extended POD approach \citep{hosseini2015sensor, discetti2018estimation}. Moreover, deep learning techniques were also applied to particle image velocimetry (PIV) for analyzing laboratory data of turbulent boundary layer \citep{cai2019dense}.
Comprehensive summaries of progress in fluid mechanics due to the introduction of  various machine learning techniques can be found in \citep{duraisamy2019turbulence, brunton2020machine}.

We have followed a different path by exploiting the universal approximation property of NNs, which together with automatic differentiation enables us to develop Navier-Stokes ``solvers" that do not require mesh generation. They are easy to implement, and can be particularly effective for multiphysics and inverse fluid mechanics problems. In particular, \citet{raissi2017physicsA,raissi2017physicsB, raissi2019physics} first introduced the concept of physics-informed neural networks (PINNs) to solve forward and inverse problems involving several different types of PDEs. This approach has also been used to simulate vortex induced vibrations in \cite{raissi2019deep} and also to tackle ill-posed inverse fluid mechanics problems, a framework called ``hidden fluid mechanics" presented in \cite{raissi2020hidden}. The flows considered in the aforementioned works are laminar flows at relatively low Reynolds numbers, described by the incompressible Navier-Stokes equations in velocity-pressure (VP) form. A fundamental question is if PINNs can simulate turbulence directly, similarly to direct numerical simulation (DNS) using high-order discretization \cite{kim1987turbulence,karniadakis2013spectral}. Another important question is if there is another formulation of the Navier-Stokes equations, e.g., in vorticity-velocity (VV) form, that may achieve higher accuracy or may be amenable to a more efficient training.

In the current study, we address the aforementioned two questions systematically by using analytical solutions for two-dimensional and three-dimensional flows and also by comparing with DNS of turbulent channel flow available at \citep{perlman2007data,li2008public,graham2016web}. In particular, we perform PINN simulations by considering two forms of the governing Navier-Stokes equations: the VP form and the VV form, and we refer to these PINNs for the Navier-Stokes flow nets as NSFnets. For the VP-NSFnet, the inputs are the spatial and temporal coordinates while the outputs are the instantaneous velocity and pressure fields. For the VV-NSFnet, the inputs again are the spatial and temporal coordinates while the outputs are the instantaneous velocity and vorticity fields. We use automatic differentiation (AD) \citep{baydin2018automatic} to deal with the differential operators in the Navier-Stokes equations, which leads to very high computational efficiency compared to numerical differentiation. However, it does not require grids, and avoids the classical artificial dispersion and diffusion errors. Furthermore, with AD we differentiate the NN rather than the data directly and hence we can deal with noisy inputs or solutions with limited regularity. There are also distinct advantages in employing both the VP and the VV formulations using PINNs. For example, to infer the pressure equation we do not use an additional Poisson pressure equation as is usually done with the traditional splitting  methods \citep{karniadakis1991high} and no data is required for the pressure as boundary or initial conditions for VP-NSFnet; the pressure is a hidden state and is obtained via the incompressibility constraint. Similarly, in the VV-NSFnet, it is easy to incorporate vorticity boundary conditions, which come in the form of constraints, directly into the loss function.

We simulate several laminar flows, including two-dimensional steady Kovasznay flow, two-dimensional unsteady cylinder wake and three-dimensional unsteady Beltrami flow, using these two types of NSFnets. We perform a systematic study using dynamic weights in the loss function for the various components following the work of \citep{wang2020understanding} to accelerate training and enhance accuracy. We also report the first results on directly simulating turbulence using PINNs. To this end, we consider
turbulent channel flow at $ \rm Re_\tau \sim $ 1,000 using primarily VP-NSFnet as the available data bases are derived based on
VP type formulations. We perform NSFnet simulations by considering different subdomains with different size at various locations in the channel and for different time intervals. In addition, we investigate the influence of weights in the loss function on the accuracy of VP-NSFnet.

The paper is organized as follows. We first introduce the NSFnets in \cref{sec:Methodology}, and present the problem set up and NSFnet simulation results for laminar flows in \cref{sec:analyticalExamples}. We then present VP-NSFnet results  for turbulent channel flow in \cref{sec:turbulentExamples}.
We summarize our findings in \cref{sec:Conclusion}.

\section{Solution Methodology}\label{sec:Methodology}
We introduce two formulations of the unsteady incompressible three-dimen\-sional Navier-Stokes equations: the velocity-pressure (VP) form and the vorticity-velocity (VV) form, as well as their corresponding physics-informed neural networks (PINNs), shown in
Fig. \ref{fig:neutral_networks}.

The VP form of the incompressible Navier-Stokes equations is:
\begin{subequations}\label{e:NS_vel_pres}
	\begin{align}
		\frac{{\partial \mathbf{u}}}{{\partial t}} + (\mathbf{u}\cdot\nabla)\mathbf{u} &= -\nabla p + \frac{1}{\text{Re}}\nabla^{2}\mathbf{u} \quad \quad  \text{in} ~\Omega ,\\
		\nabla \cdot \mathbf{u}  &= 0  \quad \quad  \text{in} ~\Omega , \\
		\mathbf{u}  &=  \mathbf{u}_{\Gamma} \quad \quad  \text{on}~ \Gamma_{D},  \\
		\frac{{\partial \mathbf{u}}}{{\partial n}} &=0 \quad\quad   \text{on} ~ \Gamma_{N},
	\end{align}
\end{subequations}
where $ t $ is the non-dimensional time, $ \mathbf{u}(\mathbf{x},t) = [u,v,w]^{T}$ is the non-dimensional velocity vector, $ p $ is the non-dimensional pressure, and $ \text{Re} = U_{ref}D_{ref}/\nu $ is the Reynolds number defined by a characteristic length $ D_{ref} $, reference velocity $ U_{ref} $ and kinematic viscosity $ \nu $. The initial and boundary conditions are required in order to solve Eq. (\ref{e:NS_vel_pres}). Here, $\Gamma_{D}$ and $\Gamma_{N}$ denote the Dirichlet and Neumann boundaries, respectively. In this study, instead of using conventional computational fluid dynamics (CFD) methods, we investigate the possibility of using neural networks (NNs) for solving the Navier-Stokes equations. In other words, the solutions of Navier-Stokes equations are approximated by a deep neural network, which takes spatial and temporal coordinates as inputs and predicts the corresponding velocity and pressure fields, i.e., $(t,x,y,z)\mapsto (u,v,w,p)$.
A schematic illustration of the PINNs for solving Eq. (\ref{e:NS_vel_pres}) is shown in Fig. \ref{fig:neutral_networks}a, which consists of a fully-connected network and the residual networks. Here, the nonlinear activation function $\sigma$ is the hyper tangent function \emph{tanh}. For the VP form, the residuals include the errors of the momentum equations and the divergence-free constraint. In order to compute the residuals of the Navier-Stokes equations $e_{\sss VP \ss 1}$ to $e_{\sss VP \ss 4}$, the partial differential operators are computed by using automatic differentiation (AD), which can be directly formulated in the deep learning framework, e.g., using ``\texttt{tf.gradients()}" in TensorFlow.

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/VP_VV.pdf}}
	\caption{A schematic of NSFnets: (a) the velocity-pressure (VP) form; (b) the vorticity-velocity (VV) form. The left part of the NN is an uninformed network, while the right part implements the VP and VV formulation using AD. We only show the operators in the right part as the NNs induced by AD of the VP and VV differential operators are too complicated and cannot be visualized even with specialized methods such as ``TensorBoard" in TensoFlow. }
	\label{fig:neutral_networks}
\end{figure}

The loss function for training the parameters of VP-NSFnet to obtain the solutions of Eq. (\ref{e:NS_vel_pres}) is defined as follows:
\begin{subequations}\label{e:loss_vp}
	\begin{align}
		L &= L_e + \alpha L_b + \beta L_i \label{e:loss_vp_L1},   \\
		L_e &= \frac{1}{N_e}\sum_{i=1}^{4} \sum_{n=1}^{N_{e}} |e_{\sss VP\ss i}^{n} |^{2} \label{e:loss_vp_Le},   \\
		L_b & = \frac{1}{N_b}\sum_{n=1}^{N_{b}}  | \mathbf{u}^{n}-\mathbf{u}_{b}^{n}|^{2} \label{e:loss_vp_Lb},    \\
		L_i & = \frac{1}{N_i}\sum_{n=1}^{N_{i}}  | \mathbf{u}^{n}-\mathbf{u}_{i}^{n}|^{2}  \label{e:loss_vp_Ld},
	\end{align}
\end{subequations}
where $ L_e $, $ L_b $ and $ L_i $ represent loss function components corresponding to the residual of the Navier-Stokes equations, the boundary conditions, and the initial conditions, respectively; $ N_b $, $ N_i $ and $ N_e $ denote the number of training data for different terms; $\mathbf{u}^{n}_{b} = [u^{n}_{b}, v^{n}_{b}, w^{n}_{b} ]^{T}$ and $\mathbf{u}^{n}_{i} = [u_{n}^{i}, v_{n}^{i}, w_{n}^{i}]^{T}$ are the given velocities for the $n$ th data point on the boundaries and at the initial time, respectively; $e_{\sss VP \ss i}^{n}$ represents the residual of the $i$ th equation at the $n$ th data point.
The weighting coefficients $ \alpha $ and $ \beta $ are used to balance different terms of the loss function and accelerate convergence in the training process. We consider the initial and boundary conditions as supervised data-driven parts, and the residual of the Navier-Stokes equations as the unsupervised physics-informed part in the loss function. We note that no data is provided for the pressure as boundary or initial conditions, which means that $p$ is a hidden state and is obtained via the incompressibility constraint without splitting the Navier-Stokes equations as done in traditional CFD methods \citep{karniadakis2013spectral}. An adaptive optimization algorithm, \emph{Adam} \citep{kingma2014adam}, is used to minimize the loss function in (\ref{e:loss_vp}).
The parameters of the neural networks are randomly initialized using the Xavier scheme \citep{glorot2010understanding}.
The solutions are obtained when the training of the NSFnet converges, i.e., the total loss function reaches some very small value.

We also propose NSFnets for the VV formulation of the Navier-Stokes equations, which is an alternative to the VP form in simulating incompressible flows; the equivalence of VP and VV formulations was proved in \citep{trujillo1999penalty,meitz2000compact}. The rotational form of the VV formation of the Navier-Stokes equations is:
\begin{subequations}\label{e:NS_vort_vel}
	\begin{align}
		\frac{{\partial \omega }}{{\partial t}} + \nabla \times (\mathbf{\omega}\times\mathbf{u}) &= - \frac{1}{\text{Re}}\nabla \times \nabla \times  \mathbf{\omega} \quad\quad   \text{in} ~\Omega, \label{e:NS_vv_a}\\
		\nabla^{2} \mathbf{u}  &= -\nabla \times \mathbf{\omega}   \quad\quad   \text{in} ~\Omega , \label{e:NS_vv_b}\\

		\mathbf{\omega}  &= \nabla \times \mathbf{u}  \quad\quad   \text{on}~ \Gamma ,  \label{e:NS_vv_c} \\

		\mathbf{u}  &=  \mathbf{u}_{\Gamma} \quad\quad   \text{on}~ \Gamma_{D} ,  \label{e:NS_vv_d} \\
		\frac{{\partial \mathbf{u}}}{{\partial n}}   &=0 \quad\quad  \text{on} ~ \Gamma_{N},    \label{e:NS_vv_e}  \\
		\nabla \cdot \mathbf{u} &= 0 \quad\quad  \text{at one point on} ~ \Gamma,  \label{e:NS_vv_f}  \\

		\mathbf{\omega}  &= \nabla \times \mathbf{u}  \quad  \text{at} ~ t=0 \quad \quad  \text{in} ~ \Omega,  \label{e:NS_vv_g}
	\end{align}
\end{subequations}
where $\mathbf{\omega}=[\omega_{x}, \omega_{y}, \omega_{z}]^{T}$ is the vorticity with three components. The boundary conditions are defined by Eqs. (\ref{e:NS_vv_c}) to (\ref{e:NS_vv_f}) and the initial condition is constrained by Eq. (\ref{e:NS_vv_g}).
Similarly, we assume that the solutions of the VV form (\ref{e:NS_vort_vel}) are approximated by a neural network, whose function can be written as $(t,x,y,z)\mapsto (u,v,w,\omega_{x}, \omega_{y}, \omega_{z})$.
The architecture of the VV-NSFnet for solving Eq. (\ref{e:NS_vort_vel}) is shown in Fig. \ref{fig:neutral_networks}b, where $e_{\sss VV \ss 1}$ to $e_{\sss VV \ss 6}$ represent the residuals of the VV formulation of the Navier-Stokes Eqs. (\ref{e:NS_vv_a}) and (\ref{e:NS_vv_b}).
The corresponding loss function of the VV-NSFnet is defined as follows:
\begin{subequations}\label{e:loss_vv}
	\begin{align}
		L &= L_e + \alpha L_b + \beta L_i      \\
		L_e &= \frac{1}{N_e}\sum_{i=1}^{6} \sum_{n=1}^{N_{e}} |e_{\sss VV \ss i}^{n} |^{2}   \\
		L_b & = \frac{1}{N_b}\sum_{n=1}^{N_{b}}  \left( | \mathbf{u}^{n}-\mathbf{u}^{n}_{b}|^{2} + |\mathbf{\omega}^{n} - \nabla \times \mathbf{u}^{n}_{b}  |^{2} + |\nabla\cdot \mathbf{u}^{n}_{b} |^{2} \right)   \label{e:loss_vv_c}  \\
		L_i & = \frac{1}{N_i}\sum_{n=1}^{N_{i}}  \left( | \mathbf{u}^{n}-\mathbf{u}^{n}_{i}|^{2} + | \mathbf{\omega}^{n} - \nabla \times \mathbf{u}^{n}_{i}   |^{2} \right) , \label{e:loss_vv_d}
	\end{align}
\end{subequations}
where $\mathbf{\omega}^n=[\omega_{x}^{n}, \omega_{y}^{n}, \omega_{z}^{n}]^{T}$ denotes the vorticity  for the $n$ th data point by NSFnet; $e_{\sss VV \ss i}^{n}$ represents the residual of the $i$ th equation at the $n$ th data point. Note that only boundary and initial values of velocity are provided in the loss function. For the vorticity term, the boundary and initial conditions are embedded in the losses (\ref{e:loss_vv_c}) and (\ref{e:loss_vv_d}) as constraints. The parameters of the neural network are also learned by using the \emph{Adam} optimizer.

Note that the weighting coefficients in the loss functions (\ref{e:loss_vp}) and (\ref{e:loss_vv}) play a very important role in the training process. However, choosing appropriate weights for NSFnets is generally very tedious. On the one hand, the optimal values of $\alpha$ and $\beta$ are problem-dependent and we cannot fix them for different flows. On the other hand, tuning the weights arbitrarily requires a trial and error procedure which is quite tedious and time-consuming. To tackle this problem, we apply the strategy of dynamic weights \citep{wang2020understanding} for choosing $\alpha$ and $\beta$ in NSFnet simulations. The idea of dynamic weights is to adaptively update the coefficients by utilizing the back-propagated gradient statistics during network training.
For a general gradient decent algorithm, the iterative formulation of the parameters of NSFnets can be expressed as:
\begin{equation}\label{e:theta_iterations}
    \theta^{(k+1)} = \theta^{(k)} - \eta  \nabla_{\theta}L_e - \eta \alpha \nabla_{\theta}L_b -  \eta \beta \nabla_{\theta}L_i,
\end{equation}
where $\theta$ denotes the parameters of the neural network, namely the weights of all the fully-connected layers, $k$ is the iteration step, and $\eta$ is the learning rate. In order to balance the contributions of different terms in Eq. (\ref{e:theta_iterations}), \citet{wang2020understanding} proposed to use a dynamic weight strategy during network training.
At each training step, e.g., $(k+1)$ th iteration, the estimates of $\alpha$ and $\beta$ can be computed by:
\begin{equation}\label{e:dyW_1}
    \hat{\alpha}^{(k+1)} = \frac{\max_{\theta}\{|\nabla_{\theta}L_e |\}}{ \overline{|\nabla_{\theta} {\alpha}^{(k)} L_b|} },  \quad
    \hat{\beta}^{(k+1)} = \frac{\max_{\theta}\{|\nabla_{\theta}L_e |\} }{ \overline{|\nabla_{\theta} {\beta}^{(k)} L_i|} },
\end{equation}
where $\max_{\theta}\{|\nabla_{\theta}L_e |\}$ is the maximum value attained by $ |\nabla_{\theta}L_e|$;
$\overline{|\nabla_{\theta} {\alpha}^{(k)} L_b|}$ and $\overline{|\nabla_{\theta} {\beta}^{(k)} L_i|}$ denote the means of $ |\nabla_{\theta} {\alpha}^{(k)} L_b| $ and $ |\nabla_{\theta} {\beta}^{(k)} L_i| $, respectively. As an alternative, we also propose the following way to estimate $\alpha$ and $\beta$:
\begin{equation}\label{e:dyW_mean}
    \hat{\alpha}^{(k+1)} = \frac{\overline{|\nabla_{\theta} L_e|}}{ \overline{|\nabla_{\theta} L_b|} },  \quad
    \hat{\beta}^{(k+1)} = \frac{\overline{|\nabla_{\theta} L_e|}}{ \overline{|\nabla_{\theta}L_i|} }.
\end{equation}
The gradients with respect to parameters of the neural network can be easily computed by AD in the deep learning framework.
Consequently, the weighting coefficients for the next iteration are updated using a moving average form:
\begin{equation}\label{e:dyW_2}
    \alpha^{(k+1)} = (1-\lambda) \alpha^{(k)} + \lambda \hat{\alpha}^{(k+1)},  \quad
    \beta^{(k+1)} = (1-\lambda) \beta^{(k)} + \lambda \hat{\beta}^{(k+1)},
\end{equation}
with $\lambda=0.1$. The strategy of dynamic weights will be applied to most of the NSFnet simulations later.

We have introduced two different formulations of NSFnets which correspond to the VP form and the VV form. We carry out several numerical experiments with NSFnets of different sizes. However, further accuracy enhancement may be possible for each case presented below using optimization in the size of architecture, the learning rate and even the optimizer, which is beyond the scope of the current work.

\section{Simulations of laminar flows}\label{sec:analyticalExamples}

In this section, we apply the proposed NSFnets to simulate different incompressible Navier-Stokes flows, including 2D steady Kovasznay flow, 2D unsteady cylinder wake and 3D unsteady Beltrami flow. We present comparisons between the VV and VP-NSFnets and investigate the influence of dynamic weights on the accuracy of the solution. Other enhancements can include the use of adaptive activation function to accelerate training \cite{jagtap2020adaptive, jagtap2019locally}, but we did not pursue this in the current work. To evaluate the performance of the NSFnet simulations, we define the relative $L_2$ error at each time step as
\begin{equation}
\epsilon_{\sss V} = \parallel \hat{V} - V \parallel_{2} / \parallel V \parallel_{2},
\label{eq:l2}
\end{equation}
where $V$ denotes the velocity components $(u, v, w)$ or the pressure $p$, and the hat represents the values inferred by NSFnets. The reference velocity and pressure are given by analytical solutions or high-fidelity DNS results. We note that to evaluate the accuracy of NSFnet solutions, we apply a shift for the NSFnet simulation results to bring the means of the pressure for reference DNS results and NSFnet results to the same value.

\subsection{Kovasznay flow}

We use the Kovasznay flow as the first test case to demonstrate the performance of NSFnets. This 2D steady Navier-Stokes flow has the following analytical solution:
\begin{equation}\label{e:Kovasenay}
\begin{aligned}
u(x,y) &= 1-e^{\lambda x} \cos (2\pi y),    \\
v(x,y) &= \frac{\lambda}{2\pi} e^{\lambda x} \sin (2\pi y),   \\
p(x,y) &= \frac{1}{2} (1- e^{2\lambda x}),
\end{aligned}
\end{equation}
where
\begin{equation*}
\lambda = \frac{1}{2\nu}- \sqrt[]{\frac{1}{4\nu^{2}}+4\pi^{2}} ,  \quad \nu=\frac{1}{\text{Re}}=\frac{1}{40}.
\end{equation*}
We consider a computational domain of $[-0.5,1.0]\times[-0.5,1.5]$. There are 101 points with fixed spatial coordinate on each boundary, such that we have 400 training data for the boundary conditions, i.e., $N_{b}=400$. For computing the equation loss of NSFnets, 2601 points are randomly selected inside the domain. There is no initial condition for this steady flow. All the NSFnets are assessed after a two-step training: we first use the \emph{Adam} optimizer for 5,000, 5,000, 50,000 and 50,000 iterations with learning rates of $1\times10^{-3}$, $1\times10^{-4}$, $1\times10^{-5}$ and $1\times10^{-6}$, respectively, then apply the limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm with bound constraints (L-BFGS-B) to finetune the results. The training process of L-BFGS-B is terminated automatically based on the increment tolerance.

For Kovasznay flow, we first investigate the influence of the neural network architecture. We employ different sizes of network by varying the number of hidden layers and the number of neurons per layer. The weighting coefficient $\alpha$ for boundary constraint is chosen as 100 for training these NSFnets. The results are summarized in Table \ref{tab:Kovasznay_results_sizes}, where each number is the best over ten independent simulations. As shown in the table, both formulations of NSFnets are able to attain the solutions with high accuracy. The relative errors are in the order of $10^{-5}$ to $10^{-3} $.
We can also observe that the performance of the NSFnets is improved as the network size increases. The VP formulation outperforms the VV form for small networks, while the VV-NSFnet provides more accurate solutions when using large networks.

\begin{table}[t]
\begin{center}
\caption{Kovasznay flow: relative $L_{2}$ errors of velocity and pressure solutions for NSFnets with different sizes ($\alpha=100$, NN size is the number of hidden layers $\times$ the number of neurons per layer).  }\label{tab:Kovasznay_results_sizes}
\begin{tabular}{c|ccc|cc}
\hline
\multirow{2}{*}{NN size} & \multicolumn{3}{c|}{VP-NSFnet} & \multicolumn{2}{c}{VV-NSFnet} \\ \cline{2-6}
& $\epsilon_{u}$ & $\epsilon_{v}$ & $\epsilon_{p}$ & $\epsilon_{u}$ & $\epsilon_{v}$ \\ \hline
$4\times50$   & 0.076\%  & 0.412\%  & 0.516\%  & 0.131\%    & 0.368\%       \\
$7\times50$   & 0.038\%  & 0.255\%  & 0.114\%  & 0.111\%    & 0.520\%       \\
$7\times100$  & 0.016\%  & 0.183\%  & 0.062\%  & 0.078\%    & 0.397\%       \\
$10\times100$ & 0.020\%  & 0.115\%  & 0.044\%  & 0.040\%    & 0.233\%       \\
$10\times200$ & 0.012\%  & 0.103\%  & 0.042\%  & 0.022\%    & 0.121\%       \\
$10\times250$ & 0.011\%  & 0.101\%  & 0.031\%  & 0.011\%    & 0.081\%       \\
$10\times300$ & 0.008\%  & 0.072\%  & 0.041\%  & 0.004\%    & 0.037\%       \\ \hline
\end{tabular}
\end{center}
\end{table}

The weighting coefficient $\alpha$ for the boundary constraint is also investigated here. In addition to letting $\alpha=100$, we also apply $\alpha=1$ and implement dynamic weights (i.e., Eqs. (\ref{e:dyW_1}) and (\ref{e:dyW_mean})) for comparisons.
In this assessment, we employ a small neural network with 4 hidden layers and 50 neurons per layer. The learning rate is decreasing during the training process as mentioned above. This strategy is consistent with the use of dynamic weights in \cite{wang2020understanding}.
The dynamic weights during the training process are displayed in Fig. \ref{fig:Kovasznay_vp_dyW}. Here $\alpha$ is initialized by 1 for the dynamic weighting strategy. We can find that the coefficient $\alpha$ oscillates and also varies due to the changes of learning rate.
The dynamic weighting strategy works similarly for VV-NSFnet. The results with dynamic weights are better than those with fixed coefficient value (i.e., $\alpha=1$ and $\alpha=100$). The resulting limit values of $\alpha$ are in the order of $10$.
The corresponding loss functions obtained by VP-NSFnet with $\alpha=1$, $\alpha=100$ and dynamic weighting strategy are illustrated in Figs. \ref{fig:Kovasznay_vp_loss}(a), (b) and (c), and for the VV-NSFnet in Figs. \ref{fig:Kovasznay_vp_loss}(d), (e) and (f).
From the curves of the training loss, we find that the \emph{Adam} optimizer is robust for the VP-NSFnet while it does not perform consistently for the VV-NSFnet. Applying two-step optimization can obtain more consistent results.
The relative $L_2$ errors of the NSFnet simulations with different weights are given in Table \ref{tab:Kovasznay_results}. For fixed weights ($\alpha=1$ and $\alpha=100$), the VP-NSFnet outperforms the VV-NSFnet for simulating Kovasznay flow. The neural network with $\alpha=100$ performs slightly better than that with $\alpha=1$ for both NSFnets.
When applying dynamic weights during network training, we can obtain more accurate solutions than the former two cases.

In order to demonstrate the effectiveness of dynamic weights, we analyze the gradients of the loss function with respect to the parameters of the NSFnets. The histograms of the back-propagated gradients ($\nabla_{\theta}L_e$ and $\nabla_{\theta} ({\alpha} L_b)$) after 10,000 iterations are shown in Fig. \ref{fig:Kovasznay_gradients}. Our goal is to balance the distributions of $\nabla_{\theta}L_e$ and $\nabla_{\theta} ({\alpha} L_b)$, thus these two terms can contribute equally to the parameter updating (i.e., Eq. (\ref{e:theta_iterations})). As shown in Figs. \ref{fig:Kovasznay_gradients}(a) and \ref{fig:Kovasznay_gradients}(e), the gradients of two different terms are unbalanced when there is no weighting coefficient for the boundary constraints (i.e., $\alpha=1$). When applying dynamic weights, the histograms of $\nabla_{\theta}L_e$ and $\nabla_{\theta} ({\alpha} L_b)$ are more consistent with each other. For the VP-NSFnet, the second formulation of dynamic weights, i.e., Eq. (\ref{e:dyW_mean}), performs better than the first one (weights given by Eq. (\ref{e:dyW_1})). However, the VV-NSFnet behaves in the opposite way for this case, as shown in Figs. \ref{fig:Kovasznay_gradients}(g) and \ref{fig:Kovasznay_gradients}(h).

\begin{table}[t]
\begin{center}
\caption{Kovasznay flow: relative $L_{2}$ errors of velocity and pressure solutions with different weights. The NN size is $4\times50$.}\label{tab:Kovasznay_results}
\begin{tabular}{c|ccc|cc}
\hline
\multirow{2}{*}{Weights} & \multicolumn{3}{c|}{VP-NSFnet} & \multicolumn{2}{c}{VV-NSFnet} \\ \cline{2-6}
& $\epsilon_{u}$ & $\epsilon_{v}$ & $\epsilon_{p}$ & $\epsilon_{u}$ & $\epsilon_{v}$ \\ \hline
$\alpha = 1$     & 0.084\%  & 0.425\%  & 0.309\%  & 0.211\%    & 1.071\%       \\
$\alpha = 100$   & 0.076\%  & 0.412\%  & 0.516\%  & 0.131\%    & 0.368\%       \\
Dynamic, Eq. (\ref{e:dyW_1})      & 0.072\%  & 0.352\%  & 0.212\%  & 0.056\%    & 0.436\%       \\
Dynamic, Eq. (\ref{e:dyW_mean})      & 0.026\%  & 0.199\%  & 0.141\%  & 0.067\%    & 0.446\%       \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{figure}[th]
\begin{center}
\includegraphics[width=9cm]{FIG/KovasznayWeights.pdf}
\caption{Kovasznay flow: dynamic weights for (a) VP-NSFnet, given by Eq. (\ref{e:dyW_1}); (b) VV-NSFnet, given by Eq. (\ref{e:dyW_1}); (c) VP-NSFnet, given by Eq. (\ref{e:dyW_mean}); (d) VV-NSFnet, given by Eq. (\ref{e:dyW_mean}). The NN size is $4\times50$.}\label{fig:Kovasznay_vp_dyW}
\end{center}
\end{figure}

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/KovasznayLoss.pdf}
\caption{Kovasznay flow: loss functions (physics loss $L_e$ and boundary loss $L_b$) obtained by (a) VP-NSFnet, fixed weight $\alpha =1$; (b)  VP-NSFnet, fixed weight $\alpha =100$; (c) VP-NSFnet, dynamic weight; (d) VV-NSFnet, fixed weight $\alpha =1$; (e) VV-NSFnet, fixed weight $\alpha =100$; (f) VV-NSFnet, dynamic weight. ``DW1" denotes dynamic weights given by Eq. (\ref{e:dyW_1}), and ``DW2" denotes dynamic weights given by Eq. (\ref{e:dyW_mean}). \emph{Adam} optimizer is used before the vertical dashed green line, and L-BFGS-B optimizer is used after that. The NN size is $4\times50$. }\label{fig:Kovasznay_vp_loss}
\end{center}
\end{figure}

\begin{figure}[th]
\begin{center}
\includegraphics[width=9cm]{FIG/Kovasznay_gradients.pdf}
\caption{Kovasznay flow: histograms of the gradients ($\nabla_{\theta}L_e$ and $\nabla_{\theta} ({\alpha} L_b$)) after 10,000 iterations during training the NSFnets. (a) VP-NSFnet, fixed weight $\alpha =1$; (b)  VP-NSFnet, fixed weight $\alpha =100$; (c) VP-NSFnet, dynamic weight (Eq. (\ref{e:dyW_1})); (d) VP-NSFnet, dynamic weight (Eq. (\ref{e:dyW_mean})); (e) VV-NSFnet, fixed weight $\alpha =1$; (f)  VV-NSFnet, fixed weight $\alpha =100$; (g) VV-NSFnet, dynamic weight (Eq. (\ref{e:dyW_1})); (h) VV-NSFnet, dynamic weight (Eq. (\ref{e:dyW_mean})).  }\label{fig:Kovasznay_gradients}
\end{center}
\end{figure}

\subsection{Two-dimensional cylinder wake}
Here we use NSFnets to simulate the 2D vortex shedding behind a circular cylinder at $\text{Re}=100$. The cylinder is placed at $(x,y)=(0,0)$ with diameter $D=1$. High-fidelity DNS data from \citep{raissi2019physics} is used as a reference and for providing boundary and initial data for NSFnet training. We consider a domain defined by $[1,8]\times [-2,2]$ and the time interval is $[0,7]$ (about one shedding period) with time step $\Delta t =0.1$. As for the training data, we place 100 points along the the $x$-direction boundary and 50 points along the $y$-direction boundary to enforce the boundary conditions and use 140,000 spatio-temporal scattered points inside the domain to compute the residuals. The NSFnets contain 10 hidden layer and 100 neurons per layer.
In addition to the default models with $\alpha=\beta=1$ and $\alpha=\beta=100$, we again implement the dynamic weighting strategy for NSFnets.
The training procedure is identical to the one we used for Kovasznay flow.

A snapshot of the vorticity contours at $t=4.0$ is shown in Fig. \ref{fig:cylinderContour}, demonstrating qualitative agreement of NSFnet inference with the DNS result.
In Fig. \ref{fig:cylinder_dyW} we present the dynamic weights of both VP- and VV-NSFnets. In this case, the weights are both initialized by the value 1. The variations of the weights correspond to the changes of learning rates. We can observe that both $\alpha$ and $\beta$ are in the order of 10 and the weights for initial conditions are larger than the weights for boundary conditions in this case. The separated terms of the weighted loss function during training are given in Fig. \ref{fig:cylinderLoss}. We employ two-step training to ensure the convergence for all the NSFnets. The relative $L_{2}$ errors of NSFnet simulations versus time are given in Fig. \ref{fig:cylinder_errors}. We see that the VV-NSFnet performs better than the VP-NSFnet, and also that applying the dynamic weights can improve the simulation accuracy for both formulations.

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/cylinderContour.pdf}
\caption{Flow past a circular cylinder: contours of the vorticity on the same contour levels at $t=4.0$: (a) reference DNS solution from \citep{raissi2019physics}; (b) VP-NSFnet, fixed weights $\alpha =\beta =1$; (c) VP-NSFnet, fixed weights $\alpha =\beta =100$; (d )VP-NSFnet, dynamic weights; (e) VV-NSFnet, fixed weights $\alpha =\beta =1$; (f) VV-NSFnet, fixed weights $\alpha =\beta =100$; (g) VV-NSFnet, dynamic weights. The dynamic weights here are given by Eq. (\ref{e:dyW_1}).}
\label{fig:cylinderContour}
\end{center}
\end{figure}

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/cylinderWeights.pdf}
\caption{Flow past a circular cylinder: dynamic weights  given by Eq. (\ref{e:dyW_1}) for (a) VP-NSFnet and (b) VV-NSFnet. }\label{fig:cylinder_dyW}
\end{center}
\end{figure}

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/cylinderLoss.pdf}
\caption{Flow past a circular cylinder: loss functions (physics loss $L_e$ and boundary loss $L_b$) obtained by (a) VP-NSFnet, fixed weights $\alpha=\beta=1$; (b) VP-NSFnet, fixed weights $\alpha=\beta=100$; (c) VP-NSFnet, dynamic weights given by Eq. (\ref{e:dyW_1}); (d) VV-NSFnet, fixed weights $\alpha =\beta=1$; (e) VV-NSFnet, fixed weights $\alpha =\beta=100$; (f) VV-NSFnet, dynamic weights given by Eq. (\ref{e:dyW_1}). \emph{Adam} optimizer is used before the dashed green line, and L-BFGS-B optimizer is used after the dashed green line. The NN size is $10\times100$. }\label{fig:cylinderLoss}
\end{center}
\end{figure}

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/cylinderErr.pdf}
\caption{Flow past a circular cylinder: relative $L_{2}$ errors of NSFnets simulations for (a) the streamwise velocity, (b) the crossflow velocity and (c) pressure. The dynamic weights here are given by Eq. (\ref{e:dyW_1}).}\label{fig:cylinder_errors}
\end{center}
\end{figure}

\subsection{Three-dimensional Beltrami flow}

The analytical solutions of the unsteady three-dimensional Beltrami flow developed by \citet{ethier1994exact} are:
\begin{equation}\label{e:Beltrami}
\begin{aligned}
u(x,y,z,t) =& -a \left[ e^{ax}\sin(ay+dz) + e^{az}\cos(ax+dy)  \right] e^{-d^{2}t},    \\
v(x,y,z,t) =& -a \left[ e^{ay}\sin(az+dx) + e^{ax}\cos(ay+dz)  \right] e^{-d^{2}t},   \\
w(x,y,z,t) =& -a \left[ e^{az}\sin(ax+dy) + e^{ay}\cos(az+dx)  \right] e^{-d^{2}t},
\\
p(x,y,z,t) =& -\frac{1}{2}a^{2} \left[ e^{2ax} + e^{2ay} + e^{2az} +2\sin(ax+dy)\cos(az+dx)e^{a(y+z)} \right.   \\
& +2\sin(ay+dz)\cos(ax+dy)e^{a(z+x)}  \\
& \left. +2\sin(az+dx)\cos(ay+dz)e^{a(x+y)} \right] e^{-2d^{2}t},
\end{aligned}
\end{equation}
where $a=d=1$ is used.
In NSFnet simulations, the computational domain is defined by $[-1,1] \times [-1,1] \times [-1,1]$ and the time interval is $[0,1]$; the time step is 0.1.
For the NSFnet training data, $31\times31$ points on each face are used for boundary conditions while a batch of 10,000 points in the spatio-temporal domain is used for the equations. The weighting coefficients of the loss function are both fixed in this case: $\alpha=\beta=100$.
The two-step optimization (\emph{Adam} and L-BFGS-B) is implemented to train the neural networks, which have the default architecture with 10 layers and 100 neurons per layer.
A snapshot of the velocity fields at $t=1$ and on the slice $z=0$ is displayed in Fig. \ref{fig:Beltrami_fields_t1_z0}.
The errors of simulation results of the two NSFnets at different time steps are presented in Table \ref{tab:Beltrami_results}, where
the relative $L_{2}$ errors of the three velocity components are given. As shown, both NSFnets can obtain accurate solutions of
the Navier-Stokes equations for the Beltrami flow, but the VV-NSFnet outperforms the VP-NSFnet.

\begin{figure}[th]
\begin{center}
\includegraphics[width=12cm]{FIG/Beltrami_fields_t1_z0.pdf}
\caption{Beltrami flow: velocity fields at $t=1$ on the plane $z=0$. (a) analytical solutions;
(b) results of VP-NSFnet; (c) results of VV-NSFnet. }
\label{fig:Beltrami_fields_t1_z0}
\end{center}
\end{figure}

\begin{table}[th]
\begin{center}
\caption{Beltrami flow: relative $L_{2}$ errors for VP-NSFnet and VV-NSFnet.  }\label{tab:Beltrami_results}
\begin{tabular}{c|cccc|ccc}
\hline
\multirow{2}{*}{$t$} & \multicolumn{4}{c|}{VP-NSFnet} & \multicolumn{3}{c}{VV-NSFnet} \\ \cline{2-8}
& $\epsilon_{u}$ & $\epsilon_{v}$ & $\epsilon_{w}$ & $\epsilon_{p}$ & $\epsilon_{u}$ & $\epsilon_{v}$ & $\epsilon_{w}$\\ \hline
$0$    & 0.067\%  & 0.059\%  &0.061\%& 0.700\%  & 0.069\%    & 0.067\%  & 0.066\%     \\
$0.25$ & 0.158\%  & 0.132\%  &0.140\%& 0.778\%  & 0.109\%    & 0.094\%  & 0.108\%    \\
$0.50$ & 0.221\%  & 0.189\%  &0.233\%& 1.292\%  & 0.118\%    & 0.119\%  & 0.132\%    \\
$0.75$ & 0.287\%  & 0.217\%  &0.406\%& 2.149\%  & 0.156\%    & 0.154\%  & 0.187\%    \\
$1.00$ & 0.426\%  & 0.366\%  &0.587\%& 4.766\%  & 0.255\%    & 0.284\%  & 0.263\%    \\ \hline
\end{tabular}
\end{center}
\end{table}
\clearpage

\section{Simulations of turbulent channel flow}\label{sec:turbulentExamples}
\subsection{Problem setup}
We simulate turbulent channel flow at $ \rm Re_\tau = 9.9935 \times 10^2$ systematically by using VP-NSFnets. We use the turbulent channel flow database \citep{perlman2007data,li2008public,graham2016web} at \url{http://turbulence.pha.jhu.edu} as the reference DNS solution. The database provides both the reference and some initial or boundary conditions for the VP-NSFnet. The DNS domain for the channel flow in the database is [0, 8$\pi$] $ \times $ [-1, 1] $ \times $ [0, 3$\pi$]; the mean pressure gradient is $dP/dx = 0.0025$.
The non-dimensional time step for DNS is 0.0013 while the online database time step is 0.0065 (five times of that for DNS). So, the time step of 0.0065 is also used for evaluating the residuals in NSFnets, i.e., five times of that for DNS. We perform NSFnet simulations by considering different subdomains with different sizes at various locations in the channel. In the first example, we place a box with size $ \sim $200 in wall-units covering a long time period. Then, we test the NSFnet simulation at a larger domain covering half the channel height. Finally, we check the influence of hyperparameters to the NSFnet simulation accuracy. We employ mini-batch to train NSFnets in this study. There are three parts in the input data corresponding to the initial conditions, the boundary conditions and the residuals of equations, respectively. Therefore, we specify the total number of iterations $n_{it}$ in one training epoch, and data in each part are evenly divided into $n_{it}$ small mini-batches. The total data in an entire mini-batch include data from each small mini-batch.

\subsection{Simulation results over a long time interval}
We first investigate if the VP-NSFnet can sustain turbulence, so  we carry out simulations covering a relatively long time interval. In this test, a subdomain at [12.47, 12.66] $ \times $ [-0.90, -0.70] $ \times $ [4.61, 4.82] (190 $ \times $ 200 $ \times $ 210 in wall-units) is considered as the simulation domain of VP-NSFnet. We perform two different simulations covering the non-dimensional time domain [0, 0.52] (81 time steps, 25.97 in wall-units) and [0, 0.832] (129 time steps, 41.55 in wall-units). Here, we define the local convective time unit of the simulation region as $ T_c^ +  = {L_x^+}/U(y)_{min} = 12.0 $. ($L_x^+$ is the size of the domain in streamwise direction.) Therefore, 25.97 and 41.55 cover more than two convective time units, i.e., $2.2T_c^+$ and $3.5T_c^+$ respectively, for this example. We use 20,000 points inside the domain, 6,644 points on the boundary sampled at each time step, together with 33,524 points at the initial time step to compute the loss function. We set the total number of iterations $n_{it} = 150$ in one training epoch. There are 10 hidden layers in the VP-NSFnet with 300 neurons per layer. The initial learning rate for \emph{Adam} decays from $ 10^{-3} $ (1,000 training epochs) to $ 10^{-4} $ (4,000 training epochs), $ 10^{-5} $ (1,000 training epochs) and $ 10^{-6} $ (500 training epochs) in the training process. The weights in Eq. (\ref{e:loss_vp_L1}) are $ \alpha $ = 100, $ \beta $ = 100. The comparisons of instantaneous flow fields between reference DNS and VP-NSFnet at $ t^+ $ = 24.67 are given in Fig. \ref{fig:contourLong}. The convergence of loss functions is shown in Fig. \ref{fig:lossLong}. The comparisons for accuracy of VP-NSFnet solutions are shown in Fig. \ref{fig:errLong}. All the simulation errors of velocity components are less than 10\%, but the relative $L_2$ error of pressure can reach 15\% or 19\%. Overall, a good VP-NSFnet simulation accuracy is obtained. These results suggest that VP-NSFnet can sustain turbulence for a long time period.

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/contourLong.pdf}
	\caption{Long time interval: comparisons of instantaneous $ z-y $ plane flow fields between reference DNS and VP-NSFnet at $ t^+ $ = 24.67: (a) reference solutions; (b) VP-NSFnet, simulation covers $2.2T_c^+$; (c) VP-NSFnet, simulation covers $3.5T_c^+$.}
	\label{fig:contourLong}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/lossLong.pdf}
	\caption{Long time interval: convergence of the VP-NSFnet simulation: (a) convergence of total loss functions and boundary loss functions; (b) convergence of residuals of governing equations.}
	\label{fig:lossLong}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/errLong.pdf}
	\caption{Long time interval: accuracy of the VP-NSFnet simulation: the solid lines and dashed lines denote simulations covering $2.2T_c^+$ and $3.5T_c^+$, respectively.}
	\label{fig:errLong}
\end{figure}

\subsection{Simulation results over a large domain}

We consider a larger domain covering half channel height in this test. We consider a subdomain at [12.47, 12.66] $ \times $ [-1, -0.0031] $ \times $ [4.61, 4.82] (about 190 $ \times $ 997 $ \times $ 210 in wall-units) as the VP-NSFnet simulation domain; and the non-dimensional time domain is set as [0, 0.104] (17 time steps, 5.19 in wall-units). We place 100,000 points inside the domain and 26,048 points on the boundary sampled at each time step, and 147,968 points at the initial time step to determine the loss function. The total number of iterations $n_{it}$ in one training epoch is taken as 150. There are 10 hidden layers in the VP-NSFnet, with 300 neurons per layer. The initial learning rate for \emph{Adam} decays from $ 10^{-3} $ (250 training epochs) to $ 10^{-4} $ (4,250 training epochs), $ 10^{-5} $ (500 training epochs) and $ 10^{-6} $ (500 training epochs) in this numerical example. The weights in Eq.  (\ref{e:loss_vp_L1}) are $ \alpha $ = 100, $ \beta $ = 100. Comparisons of instantaneous flow fields between reference DNS and VP-NSFnet at the final simulation time step are shown in Fig. \ref{fig:contour1000}. The convergence and accuracy of VP-NSFnet solutions are shown in Fig. \ref{fig:err1000}. All the simulation errors of velocity components are less than 10\%, but the relative $L_2$ error of pressure can reach 17\%. In such a large domain, complex interactions between different scales of eddies occur, and this domain covers the whole range including the law of the wall, the viscous sub-layer, the buffer layer, the log-law region and the outer layer \citep{pope2000turbulent}. However, VP-NSFnet can still get very accurate solutions. The results also indicate that the relative $L_2$ errors of the wall-normal and spanwise velocities are much higher than that of streamwise velocity, i.e., nearly an order higher. This is caused by larger amplitude of the streamwise velocity than the other two velocity components, also nearly an order higher, as shown in Fig. \ref{fig:err1000}. A proper normalization with careful tuning of anisotropic weights may result in a more balanced accuracy.

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/contour1000.pdf}
	\caption{Large domain: comparisons of instantaneous flow fields between reference DNS and VP-NSFnet at $ t^+ $ = 5.19 covering half channel height: (a) reference solutions, $ x-y $ plane, $z^+ = 0$; (b) VP-NSFnet, $ x-y $ plane, $z^+ = 0$; (c) reference solutions, $ z-y $ plane, $x^+ = 0$; (d) VP-NSFnet, $ z-y $ plane, $x^+ = 0$.}
	\label{fig:contour1000}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/err1000.pdf}
	\caption{Large domain: convergence and accuracy of VP-NSFnet covering half channel height: (a) convergence of loss function; (b) relative $L_2$ error.}
	\label{fig:err1000}
\end{figure}

\subsection{Influence of weights}

In the above numerical experiments for turbulent channel flow, all the weights are manually tuned to obtain satisfying results. In this section, we investigate the influence of weights, especially the dynamic weights, to the VP-NSFnet simulation accuracy.
The formulation of Eq. (\ref{e:dyW_1}) is applied to the loss function of VP-NSFnet. However, different from Eq. (\ref{e:dyW_1}), a normalization factor $\gamma$ is taken into account in this section. Hence, the dynamic weights for the turbulence simulation can be expressed as: \begin{equation}\label{e:dyW_dns}
    \hat{\alpha}^{(k+1)} = \frac{\max_{\theta}\{|\nabla_{\theta}L_e |\}}{ \gamma\overline{|\nabla_{\theta} {\alpha}^{(k)} L_b|} }.
\end{equation}
We consider a subdomain of [12.53, 12.59] $ \times $ [-1, -0.9762] $ \times $ [4.69, 4.75] (about 60 $ \times $ 24 $ \times $ 60 in wall-units) as the VP-NSFnet simulation domain; the non-dimensional time domain is set as [0, 0.104] (17 time steps, 5.19 in wall-units). For this small domain, the initial velocity values are not used in the loss function, i.e., $\beta = 0$, but instead we can learn them. There are 2,000 points inside the domain and 1,100 points on the boundary sampled at each time step to determine the loss function. We take the total number of iterations $n_{it}$ of one training epoch as 10. There are 5 hidden layers in the VP-NSFnet, with 200 neurons per layer. In all the examples, the initial learning rate for \emph{Adam} decays from $ 10^{-3} $ (5,000 training epochs) to $ 10^{-4} $ (5,000 training epochs), $ 10^{-5} $ (25,000 training epochs) and $ 5 \times 10^{-6} $ (25,000 training epochs).
We set five different strategies to take different weights for the boundary data in this experiment. In the first two strategies, we use fixed weights of $\alpha = 1$ and $\alpha = 100$. Then, we use the dynamic weights given by Eq. (\ref{e:dyW_dns}) with different normalization factors, i.e., $\gamma = 1$, $\gamma = 5$ and $\gamma = 10$. The comparisons of instantaneous flow fields ($ z-y $ plane) between reference DNS and VP-NSFnet simulations at $ t^+ $ = 5.19 and $x^+ = -12.27$ with various weights are shown in Fig. \ref{fig:hyperContour}.
The evolution of dynamic weights in the training process is shown in Fig. \ref{fig:hyperWeights}.
The convergence of the VP-NSFnet simulation with various hyperparameters is shown in Fig. \ref{fig:hyperLoss}.
The accuracy of the VP-NSFnet simulation with various hyperparameters is shown in Fig. \ref{fig:hyperErr}. It is noted from Fig. \ref{fig:hyperContour} that there exists large discrepancy between the reference solution and the VP-NSFnet solution with fixed weight $\alpha=1$, thereby its accuracy is of low level. Therefore, the VP-NSFnet simulation accuracy with fixed weight $\alpha=1$ is not shown in Fig. \ref{fig:hyperErr}.
Overall, the best result for this turbulence simulation is obtained when applying dynamic weights with $\gamma=5$. A dynamic variation of the weights can improve the performance of VP-NSFnet.

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/hyperContour.pdf}
	\caption{Influence of weights: comparisons of instantaneous flow fields ($ z-y $ plane) between reference DNS and VP-NSFnet at $ t^+ $ = 5.19 and $x^+ = -12.27$: (a) reference solutions; (b) VP-NSFnet, fixed weight $\alpha = 1$; (c) VP-NSFnet, fixed weight $\alpha = 100$; (d) VP-NSFnet, dynamic weight with normalization factor $\gamma = 1$; (e) VP-NSFnet, dynamic weight with normalization factor $\gamma = 5$; (f) VP-NSFnet, dynamic weight with normalization factor $\gamma = 10$. All the dynamic weights here are given by Eq. (\ref{e:dyW_dns}).}
	\label{fig:hyperContour}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=6cm] {FIG/hyperWeights.pdf}
	\caption{Influence of weights: dynamical weights with different normalization factors versus training epochs. All the dynamic weights here are given by Eq. (\ref{e:dyW_dns}).}
	\label{fig:hyperWeights}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/hyperLoss.pdf}
	\caption{Influence of weights: convergence of the VP-NSFnet simulation: (a) VP-NSFnet, fixed weight $\alpha = 1$; (b) VP-NSFnet, fixed weight $\alpha = 100$; (c) VP-NSFnet, dynamic weight with normalization factor $\gamma = 1$; (d) VP-NSFnet, dynamic weight with normalization factor $\gamma = 5$; (e) VP-NSFnet, dynamic weight with normalization factor $\gamma = 10$. All the dynamic weights here are given by Eq. (\ref{e:dyW_dns}).}
	\label{fig:hyperLoss}
\end{figure}

\begin{figure}
	\centerline{\includegraphics [width=12cm] {FIG/hyperErr.pdf}
	\caption{Influence of weights: accuracy of VP-NSFnet for various weights: (a) to (d) relative $L_2$ errors of $u$, $v$, $w$ and $p$.}
	\label{fig:hyperErr}
\end{figure}

\section{Summary}\label{sec:Conclusion}
In this study, we explored the effectiveness of PINNs to directly simulate incompressible flows, ranging from laminar flows to turbulent channel flow. We have formulated NSFnets based on two different forms of the governing Navier-Stokes equations: the velocity-pressure (VP) form and the velocity-vorticity (VV) form. The spatial and temporal coordinates are the inputs of the PINNs, and the instantaneous velocity and pressure fields are the outputs for the VP-NSFnet; similarly, the instantaneous velocity and vorticity fields  are the outputs for the VV-NSFnet. We used automatic differentiation to represent all the  differential operators in the Navier-Stokes equations; then the equations can be formulated by the neural networks. We regard the initial and boundary conditions as supervised data-driven parts, and the residual of the Navier-Stokes equations as the unsupervised physics-informed part in the loss function of PINNs. We note that no data was provided for the pressure as boundary or initial conditions for VP-NSFnet, which was a hidden state and was obtained indirectly via the incompressibility constraint without splitting the equations. Convergence of NSFnets was monitored using the total loss function as well as the individual loss functions. We simulated several laminar flows, including two-dimensional steady Kovasznay flow, two-dimensional cylinder wake and three-dimensional Beltrami flow using the two forms NSFnets. We also carried out a study on the influence of weights in the various contributions to the loss function.  We found that for laminar cases, the VV-NSFnet achieves better accuracy than the VP-NSFnet, and that a dynamic variation of the  weights can improve the performance of both NSFnets.

In addition, we explored the possibility of simulating turbulent channel flow at $ \rm Re_\tau \sim $ 1,000 using NSFnets. We performed NSFnet simulations by considering different subdomains with different sizes at various locations in the channel and for different time intervals. Established DNS databases provided the proper initial and boundary conditions for the NSFnet simulations. We obtained good agreement between the DNS results and the VP-NSFnet simulation results upon convergence of the loss function. The long time period simulation suggests that NSFnets can sustain turbulence with errors bounded at reasonable levels. We also investigated the use of dynamic versus fixed weights and demonstrated how dynamic weights could further enhance the accuracy of VP-NSFnet. Unlike the VP-form, attempts to train the VV-NSFnet failed to provide satisfactory  convergence of the loss function for the governing equations. For $\alpha=50,000$ we obtained reasonable accuracy, with the loss function for boundary conditions converging to small values but the residual for the governing equations remained very large. This may be related to the fact that the data in this case  are obtained from DNS data bases with a VP formulation, and hence the boundary conditions derived may not be so accurate but rather inconsistent with the VV form of the governing equations. We plan to revisit this issue in future work.

The current study for modeling turbulence using NSFnets is the first attempt to evaluate the PINN performance, and while the first results are encouraging, the broader question is if PINNs can provide sufficient accuracy to sustain turbulence with non-stochastic boundary conditions but rather the simple zero Dirichlet and periodic boundary conditions employed for the entire domain in spectral type simulations of turbulence. To address this question expeditiously, the efficiency of PINNs has to be improved significantly, including the development of a multi-node GPU code that can accelerate significantly the training process. Such speed up can be further enhanced by using adaptive activation functions as demonstrated in \cite{jagtap2020adaptive,jagtap2019locally}.
Moreover, a further study is needed to derive the proper normalization procedure for the three different velocity components so that we can obtain uniform accuracy, as in the current study the streamwise component is inferred with an order of magnitude higher accuracy compared to the crossflow velocity components when the streamwise velocity is of an order higher than the crossflow velocity.

\section*{Acknowledgements}
X. Jin and H. Li would like to acknowledge funding from the National Natural Science Foundation of China (NSFC) (Grant No. U1711265). S. Cai and G.E. Karniadakis would like to acknowledge funding from DARPA-AIRA grant HR00111990025 and the DOE grant DE-AC05-76RL01830.

redpajama_set_name: RedPajamaArXiv


Cleansing crisis is also known as a healing crisis. This occurrence is common when one takes our Herbal Detox program- Yeast-Para Control and Cleansing-Balance Tea. The occurrence is also common for those who simply start a yeast-free nutritional regimen containing no sugar, no dairy, no wheat, no yeasts, no caffeine, no alcohol, no nicotine nor chemicals.
The reaction occurs when your body tries to eliminate toxins. The more toxic your body systems are, the more severe the detoxification, or healing crisis.
I normally alert my clients of the following: Whatever symptoms you are experiencing now could be worse. Additional symptoms you don't experience currently may also appear.
Some people who are not familiar with the concept of cleansing crisis may jump to the conclusion that they are allergic to Dr. J's herbs or holistic treatment programs and rush to their MD's for more prescriptions of antibiotics or cortisone. The fact is these are signs that the detox program is working effectively to remove the toxins from your body, and thereby allows your body's innate intelligence to heal you.
Cleansing reactions can occur immediately, or within several days, or even several weeks after you start our Herbal Detox, Hormonal Regeneration, Weight Management, and/or yeast-free nutritional programs. Going through cleansing crises is like peeling an onion layer by layer. Cleansing crises are like waves – they come and go. Sometimes "tidal waves" can occur, and you might feel like you were run over by a truck! Committed clients have told me that they "feel great about feeling miserable"! Some even take a few days off from work and rest, so that it's easier for them to go through each cleansing crisis.
Sometimes discomfort during the healing crisis is of greater intensity than when you were developing and enduring the chronic disease. This is why there may be a brief flare-up in your symptoms. Often the crisis will come after you are feeling your best. I called it the "honey moon period". After the honey moon period, cleansing crises occur because your body dumps toxins into the blood stream for elimination. When the stored wastes are in a free-flowing state, a variety of symptoms may flare up.
The healing crisis will usually bring about past conditions. People often forget the diseases or injuries they have sustained in the past, but are usually reminded of during the crisis.
How does one manage a cleansing crisis?
Basically there are two ways: sharp, short pain, or dull, long pain. It does not matter which way you chose, the same amount of toxins need to be removed.
Most of my clients drink a substantial amount of Cleansing-Balance Tea to flush out toxins in the liver and kidneys during the cleansing crisis. One tea bag of Cleansing-Balance Tea makes one pot: 6 cups (8 oz. each cup). Some of my clients drink two pots a day during the cleansing crisis. Braver clients with situations such as drug addiction or Irritated Bowel Syndrome (IBS) increase the dosage of our herbal formulas to hasten the cleansing effects. If you need to work and/or perform a certain amount of daily chores, then just reduce the dosage of our herbal formulas and go through a milder, but longer cleansing period.
You also can choose to stop the cleansing crisis and go back to the same unhealthy lifestyle. However, I have news for you: it's not uncommon that these kinds of clients come back to me after they have developed life-threatening diseases such as cancer, tumors, heart disease, etc. By that time, they would need to spend more resources, time and effort to go through the cleansing crisis, all the while living under the shadow of death.
The choice is yours. With each choice you make, you need to prepare for the consequences associated with it. As a helping professional, my job is to simply help my clients live or die more comfortably.


Relaxing days spent outdoors require hydration. Created with a freeze-dried coconut water base, this tea will keep your body happy all day. Every sip brings light and refreshingly sweet notes of the tropics with bright pineapple. A base of organic green rooibos adds a fresh herbal note.
Excellent! Love the subtle flavor. Made a double batch. Used slightly less ice water. Double batch made about 6 1/2 cups. Will be ordering more.
4 Stars I love pineapple!
I loved this iced tea! Cool, refreshing, relaxing, what else can I say? I love most things pineapple and this did not disappoint!
I enjoy this tea. The flavor is a bit subtle and I would enjoy a bit stronger flavor, but it's very refreshing and light.
Again a perfect ice tea. Just enough for the summer a refreshing drink. If you want more umph "caffeine" use a few black tea bags when preparing it. When I know the boys are going to be by thats how I make it. They LOVE it too.
This is a great, mildly sweet, tropical iced tea. I prefer to blend teas for iced tea, so I pair this with a white tea, and pineapple tea. It's not too strong like the green tea version. Perfect for several glass fulls.


If your work involves handling small or fine parts, it's important that your gloves are comfortable and breathable to reduce hand fatigue and enable precision handling for extended periods. The Marigold Industrial PX140 Lightweight Multi-Purpose Work Gloves feature a lightweight, stretchable liner for an optimised fit and feel. A polyurethane-coated palm provides excellent abrasion resistance and grip, while a ventilated back ensures that the hand can breathe naturally.
The PX140 are designated Category II gloves, meaning they are suitable for medium-duty applications and intermediate levels of risk.
The Marigold Industrial PX140 are supplied in five different sizes, meaning they should be suitable for most users. Please measure the circumference of your hand around the knuckles (excluding the thumb) with your hand open and your fingers together, and the length of your hand from the tip of your middle finger to your wrist.


Delivery time on request ?
This item is not available from stock, but may be available with our supplier. In most cases, the item can still be delivered quickly. Please contact our customer service for the best price and delivery time.
Live and loud, Electromatic Center Block Jr guitars are designed for that pure and powerful Gretsch sound at high volume. Painstakingly crafted with essential Gretsch sound, style and playability, these all-new models deliver the fearless and electrifying authenticity that has set Gretsch apart for decades. It's the sound of honor itself—play proud.
The G5655TG Electromatic Center Block Jr Single Cut with Bigsby and Gold Hardware features a smaller-sized body with a chambered spruce center block for dynamic resonance, high gain power and lightweight comfort.
Offering excellent playability, the 12" radius laurel fingerboard hosts 22 medium jumbo frets and NeoClassic thumbnail inlays. Its dual Black Top Broad'Tron humbucking pickups deliver bold, dynamic and articulate tone with enhanced sonic punch.
This guitar boasts stage-stealing style with a stunning finish and gleaming gold hardware including the Bigsby B70 vibrato tailpiece, anchored Adjusto-Matic bridge and G-arrow knobs.
• Neck Shape: Thin "U"
Gretsch Barstool Since 1883 24"
Gretsch Barstool Since 1883 30"


text:
#include <linux/types.h>

/* platform data for the PL061 GPIO driver */

struct pl061_platform_data {
	/* number of the first GPIO */
	unsigned	gpio_base;

	/* number of the first IRQ.
	 * If the IRQ functionality in not desired this must be set to NO_IRQ.
	 */
	unsigned	irq_base;

	u8		directions;	/* startup directions, 1: out, 0: in */
	u8		values;		/* startup values */
};


text:
Q: AttributeError: 'tuple' object has no attribute 'value' PYTHON TO EXCEL I am currently trying to write data to an excel spreadsheet using Python3 and openpyxl. I have figured out how to do it when assigning one single value, but for some reason when I introduce a For loop it is giving me an error. This program will eventually be used to filter through a python dictionary and print out the keys and values from the python dictionary. For now, I am just trying to create a loop that will input a random integer in the spreadsheet for every key listed in the dictionary (not including nested keys). If anyone can help me determine why this error is coming up it would be much appreciated. Thanks in advance!
# Writing Dictionary to excel spreadsheet
import openpyxl
import random

wb = openpyxl.load_workbook("ExampleSheet.xlsx")
sheet = wb.get_sheet_by_name("Sheet1")

sheet["B1"].value = "Price" #This works and assigns the B1 value to "price" in the spreadsheet

my_dict = {'key': {'key2' : 'value1', 'key3' : 'value2'} 'key4' : {'key5' : 'value3', 'key6' : 'value4'}} #an example dictionary

n = len(my_dict)
  for i in range(0,n):
    sheet['A'+ str(i)].value = random.randint(1,10) #This does not work and gives an error

wb.save('ExampleSheet.xlsx')

OUTPUT >>> AttributeError: 'tuple' object has no attribute 'value'

A: The first column of pyxl, is one based, so if you modify your loop to go over range(1,n) your issues should be resolved

A: Using .format(i) instead of string + str(i) in ur code may work well!
BTW, ur var my_dict get an error .
eg:
for i in range(10):
    sheet['A{}'.format(i)].value = 'xx'

redpajama_set_name: RedPajamaStackExchange


text:
Party to celebrate Susan Brown in retirement is June 30
After facilitating many software upgrades, Susan Brown, the manager of ATUS Software Services, will upgrade her status to "Retired" on July 1. The Western Washington University community is invited to celebrate her 21 years at Western at an informal gathering from 9 to 10:30…
Campus tech group to discuss Creative Cloud, workflow Nov. 5
The Learning Technology Group meets from 3 to 4 p.m. Wednesday, Nov. 5, to discuss Creative Cloud.
The presentation will be delivered and accessible via the Big Blue Button web conferencing tool available within Canvas. The session will be recorded for later access via the Creative Cloud…
Adobe staff to offer free training in use of Creative Cloud Nov. 13
Adobe Creative Cloud software is now available at Western Washington University.
To help the campus community get the maximum benefit from this software, Adobe representatives will be on campus Thursday, Nov. 13, to present some of the most popular new features.
Adobe to be on campus Aug. 26 to help faculty, staff learn Creative Cloud
Adobe Creative Cloud software is now available at Western Washington University. To help Western employees get the maximum benefit from this software, Adobe representatives will be on campus next Tuesday, Aug. 26, to present some of the highlights and new features. Registration is appreciated.…
Microsoft Office 2013 available for home use
Microsoft Office Pro Plus 2013 for Windows is now available to Western Washington University faculty and staff under the Microsoft Home Use Program for a $9.95 transaction fee. Email software.services@wwu.edu or call 360-650-3159 to obtain the program code and download link. Each employee using…
Students can get free Microsoft software, including Windows 8
A special trial program has been established with Microsoft Corporation to allow all current Western Washington University students to receive Microsoft Windows operating system upgrades, including Windows 8, and the complete Office Professional Suite (including Word, Excel, PowerPoint, etc.)…
Subscribe to Software Services


Minor keys are awesome. They sound dark and dramatic, and many musicians over the centuries have made great music with them. They're also kind of a hassle to figure out. As I promised in a previous post, we're going to delve into minor scale harmonization and their harmonic areas. If you are unfamiliar with the makeup of minor scales, please check out Smitch's explanation here.
*Note: Common practice is to compare the minor scale to the major scale. Therefore chords like the III chord in minor are referred to as ♭III, being a minor third above the root.
Literally the exact same series shifted over. We could then assume that the chord functions – which harmonic areas they may fit into – stay the same. But this isn't the case. There are three different forms of minor scales that are commonly used in tonal music: natural, harmonic and melodic. The differences between the three of them are fairly easy to spot.
The problem is that now we have different notes and thus, different chords depending on what form of minor we're using. For example, the v chord which was previously composed of scale degrees 5 ♭7 2 is now 5 7 2. That raised scale degree 7 changes the v chord to a V chord, or minor to major. To understand the full implications of this, let's talk about the development of these scales in western music.
Another note: The following section is a loose summary of generations of musical evolution. The way music theory was taught and used was way different back then. The technical explanations still stand. If any musicologists have complaints, send them to the Guitar-Muse.com editors. I probably won't read them.
These old school dudes came to the conclusion that the distance between the ♭6 and the raised 7 (an interval known as an augmented 2nd) was kind of tough to sing. To remedy this situation, they decided to raise scale degree 6 as well. This made certain melodic lines easier to sing and thus the modern term "melodic" minor. It was common practice for centuries to only raise these scale degrees on the way up. Descending the scale meant playing your regular old ♭6 and ♭7. This was so common, that even today people refer to a melodic minor scale played the same way in both directions as jazz melodic – jazz being a very young style compared to western classical music.
Take careful note of the differences between the chords that occur in natural, harmonic and melodic minor scales. It's a good idea to review your scale fingerings, as well as finding a few comfy fingerings for some of the harder chords.


I'm 17 years old and work with kids 6 months to 12 years. I can cook and clean. Please consider me! Thanks!
Searching for Boise Mother's Helper Opportunity! I love Kids!
Boise Baby Sitter/care giver/children with disability needs. Looking For work.


text:
Storage software revenue stays aloft
Companies' desire to squeeze more out of storage gear pushes storage software revenue to a 17 percent rise over last year, IDC says.
September 13, 2004 11:41 AM PDT
Growing demand for more efficient storage helped push second-quarter revenue from storage software to a 16.9 percent rise over last year, according to a new IDC study.
But compared with the last quarter, the growth in worldwide revenue was negligible, the research firm said in the report, released on Monday. As in the , the sector netted $1.85 billion in the three months to June.
The increase registered by the storage software segment was far more significant than that seen by the hardware sector. According to another recent IDC report, worldwide shipments of external disk storage systems grew only by 8 percent in the second quarter.
The most considerable second-quarter lift was reserved for device and SAN management tools, which saw a revenue increase of 36 percent from a year ago. File system software was up 18.5 percent, while storage replication rose 13.5 percent. Back-up and archive software saw an increase of 9.2 percent.
For the first time, storage resource management emerged as the largest of the storage software markets, as companies saw the need to support larger and more complex storage networks, Bill North, IDC's storage software research director, said in a statement.
EMC and Veritas, which registered 30.5 percent and 23.4 percent year-over-year growth, respectively, held on to the top two positions in the storage software market. EMC, with a 32.5 percent market share, is far ahead of Veritas, which has 22.6 percent, according to the IDC report. Third-place IBM and fourth-place Hewlett-Packard left Computer Associates International in the fifth position.
We drowned AirPods, Powerbeats Pro and Galaxy Buds: We sprayed them, dunked them and even put them through the wash to find out which one of these three wireless earphones can handle the most water.
Discuss: Storage software revenue stays aloft


south indian jewelry temple jewellery google search more set for wedding.
custom bookshelf picture of installation builders.
modern bread boxes box cabinet vintage all metal mid century steel style home improvement license test questions.
carve wood bowl a selection of finished natural bowls by later she will add lacquer home improvement license ct.
white hanging planter modern apartment 2 pack geometric wall decor container great for succulent baskets uk.
baby girl onesie previous funny outfits.
science teeshirts celebrate women in t shirt march for amazon.
viking baby toddler crochet hat with beard kids helmet shower.
personalized yard flags birthday boy garden flag custom.


Matt August is a theatre director specializing in new plays and musicals, Shakespeare, comedies and holiday classics.
Matt has enjoyed directing and developing work on Broadway, at theatres all over the country, for LORT, non-profit and commercial producers. He was recently a finalist for the 2016 TCG Alan Schneider Directing Award for mid-career directors. His productions have been: recognized with nominations and awards from the Ovation, Helen Hayes, Mac, Bay Area Critics, Broadway World, and Australia's Helpmann Awards. They have been showcased at The Macy's Thanksgiving Day Parade, on Good Morning America, The Today Show, The Wendy Williams Show and even performed at The White House. They have appeared on Year End Top-Ten lists for the LA Times, San Francisco Chronicle, Huffington Post, Arizona Republic, Oakland Tribune, Tuscon Sun, San Jose Mercury News and NPR/KQED.
Matt has been awarded fellowships and residencies from the Old Globe Theatre, Drama League, Oregon Shakespeare Festival, Acting Company, Robert Wilson's Watermill Center, the Juilliard School, San Francisco's Zen Center as well as directing and assisting at many other regional theatres.
He is a protégé of theatre director Jack O'Brien, working with O'Brien on Broadway for four years ('00-'04). Earlier in August's career, he worked with many other directors ranging from Gordon Davidson, John Rando and Nicholas Martin to Stephen Wadsworth, Libby Appel and Robert Wilson.
As an active member of the Society of Directors and Choreographers, he is currently on the Los Angeles Steering Committee, the LORT Negotiating Committee and the L.A. Intimate Theatre Contract Negotiating Team.
August has mentored several upcoming directors through the Drama League and SDC Foundation, guest taught/lectured at universities and colleges throughout the country and served on the Drama League Directing Fellowship selection committees and SDC Board Nominating committees. He also sits on the Artistic Advisory Board for the Independent Shakespeare Company in Los Angeles.
August holds an MFA from California Institute of the Arts and splits his time between Los Angeles and New York City.
Utah's Deseret News said his production of In the Heights "Sizzles…[It] is not to be missed." and later that August has made Shakespeare cool" about Much Ado About Nothing.
The San Francisco Chronicle described his work as "hilarious and moving…imaginatively staged" (Fourth Messenger) and "captivating and inventively eclectic" (Baby Taj).


It's that time again for the Heartfelt Creations Alumni Hop. For my card this month, I used the new release Water Lily stamps, and Water Lily Dies.
My leaves are also HFC; Open Leaf Cling Stamp Set, with Classic Leaf Dies.
That's it for me this month. The following is a list of the participating Designers. Remember to comment along the way as there is a random winner chosen for the $25 gift certificate!
OMG Vicki, this is STUNNING!!! I have had my eye on that set, it's so cute. Beautiful flowers here, now I NEED it!!!
I am such a huge fan of your fresh, modern take on vintage! The jewel tones on this card are truly breathtaking! Gorgeous make!
This is amazing. You really showcased that pretty paper and I love the colors you chose to work with! Well done!
Such a lovely one!! Love the colors you have added!! So beautiful!
This is so beautiful. I love the colors of it too.
What pretty colors, this looks lush and tropical!!
Beautiful colors and I like the striped border.
Such beautiful flowers. I love your coloring.
Beautiful card and pretty coloring!
Such a pretty card. Loved the colors and the black and white panel which add texture and breaks out of the color schema.
What a stunning card. I love the colors you used.
I like the colors.... my favorite!!!
Oh, and I think THIS is amazing! Love your colors and those flowers are lovely! Hugs!
Love the coloring and clean lines of your card.
beautiful card! Love the vibrant colors of the flowers!!
What beautiful patterned paper, Vicki and I love the B&W stripes to go with it!!! The flowers are simply beautiful and who would not love to receive a card with such a sweet sentiment!!!! Fabulous, my friend!!!
Love the cards, of course anything Hearfelt creation I love.


An extensive selection of carefully chosen and high quality works of Art for sale.
Diachroniki Gallery specialises in art with the theme Cyprus of yesterday and today. In its two galleries in the centre of the old town of Nicosia you will find a wide selection of high quality works of art.
We sell paintings, icons, sculpture, object d'art, antique prints, maps, jewellery, watches, limited editions and monotypes. In addition we stock over 1500 different reproductions with scenes from old and contemporary Cyprus.
Diachroniki Gallery also organises regularly exhibitions, lectures, documentaries on art, book presentations and seminars.
Diachroniki Gallery is open daily from 10:00 – 19:00.


HP Laptop Screen Lines – A Damaged Screen is usually the problem.
When you look at your laptop screen and it displays lines usually horizontally or vertically then that is a sign that the screen itself is damaged. This doesn't happen through wear and tear but can happen usually if the screen has taken an impact. Even if you think the knock was minimal that can still be enough to cause the fault.
The only solution is to replace the whole screen, they cannot be repaired if they have been damaged. At Laptop Power UK we stock a wide range of screens in various sizes from 8.9" all the way up to 18.4". Whether you will take the screen to a repair store or you will attempt to fit the part yourself we have the ideal replacement HP laptop Screen for you. If you want to fit the screen yourself we provide detailed instructions on our website which gives you a step by step guide as how to do it. Our screens are sent with a secure next working day courier in all instances and come securely packaged. When you need a Replacement HP Laptop Screen take a moment to visit www.laptop-power.co.uk .
This entry was posted on Sunday, December 19th, 2010 at 6:50 pm. You can follow any responses to this entry through the RSS feed.


Trust the Heritage Group, LLC with all your REO needs.
We work with all lenders both large and small.
We are a privately owned real estate agency centrally locate in Connecticut with state wide coverage. We handle all phases of asset disposition and REO's directly in house. Our exceptional agents understand the unique process, requirements and challenges associated with REO's. Having an in house construction entity partner allows us to offer complete asset renovation and repair services to our clients.


While the rest of the economy struggles with worker layoffs, independent insurance agencies are reportedly facing a very different problem: filling their jobs with knowledgeable workers. As a result, more firms facing a shortage of qualified workers, have shifted from buying talent, to building talent.
That is one of the top two trends identified by Business Management Group's (BMG's) 2003-2004 Non-Producer Compensation & Benefits Study, based on responses from more than 400 agencies and brokerage firms nationwide. BMG, a subsidiary of The Hartford Financial Services Group, is a management consulting firm specializing in the insurance industry.
The second major trend is that a growing number of agencies are paying closer attention to total compensation, spurred on by the growing cost of healthcare coverage.
But this approach to cost containment isn't enough. The tight labor market has driven up mean salaries for operations and sales managers by 18 percent since the last survey in 2001. This is also true for Employee Benefits CSRs, where mean salaries have reportedly increased 17 percent in that two-year period.
According to the survey, the trend in compensation has clearly shifted towards variable pay related to business results and meeting individual objectives.
In 1999, surveyors found that 47 percent of participating agencies offered incentive plans to managers, while today that number has grown to 82 percent. The percentage of companies that offer incentives to non-manager staff has held steady at a significant 74. Today, 56 percent of the participants reward managers based on specific performance objectives, and over 18 percent offered long-term incentive plans to managers.
Most participants in the United States indicate their planned compensation increases for 2003 will be between 3.0 and 5.7 percent, with the Southwest accounting for the highest anticipated increase, and the Northeast the lowest.
Projected increase rates for 2003 are slightly higher than 2002 actual increases, reportedly indicating agents and brokers' concerns about a lack of available talent and the need to remain competitive.


Come Arredare Una Veranda Tendenza Di Arredo Terrazze E Verande - Prestobarandgrill.com.
Prestobarandgrill.com (https://www.prestobarandgrill.com) - Come Arredare Una Veranda Tendenza Di Arredo Terrazze E Verande image uploaded on Sunday, December 16 2018 by . this picture size is 1444 x 960. to download this image, you can hover on image above then right click and save as.
Gellery Of Come Arredare Una Veranda Tendenza Di Arredo Terrazze E Verande.


Iris Hantverk's Everyday Dish Brush is made from beech wood and horse hair. Horse hair is soft and gentle and effectively cleans without scratching porcelain. You can replace the brush with the refill.
Coastal Scandi in Wadebridge, United Kingdom.