text:
\section*{Appendices}
\renewcommand\thesubsection{\Alph{subsection}}
\renewcommand\thesubsubsection{\thesubsection.\arabic{subsection}}
\input{sec-appendix}

\end{document}
\endinput

\subsection{Algorithm and Complexity}

\subsection{Algorithm and Complexity Analysis} \label{app.alg}
\stitle{Algorithm.}
We present the algorithm for prompt design and tuning of \method{GraphPrompt}{}\ in Alg.~\ref{alg.prompt}. In line 1, we initialize the prompt vector and the objective $\ensuremath{\mathcal{L}}_{\text{prompt}}$.
In lines 2-3, we obtain the node embeddings of input graphs based on the pre-trained GNN. In lines 5-13, we accumulate the loss for the given tuning samples. In particular, in lines 5-6, we design the prompt for the specific task $t$. In lines 7-8, we calculate the subgraph representation for each class prototype. Then, in lines 9-13, we calculate and accumulate the loss and get the overall objective. Finally, in line 14 we optimize the prompt vector by minimizing the objective $\ensuremath{\mathcal{L}}_{\text{prompt}}$.

\begin{algorithm}[h]
\small
\caption{\textsc{Prompt Design and Tuning}}
\label{alg.prompt}
\begin{algorithmic}[1]
    \Require Graphs set $\ensuremath{\mathcal{G}}=\{G_j|j=1,2,\ldots\}$, task $t$-specific subgraphs set $\ensuremath{\mathcal{S}}=\{S_{t,x}|x=1,2,\ldots\}$,
    labeled set $\ensuremath{\mathcal{D}}=\{(x_i,y_i)|i=1,2,\ldots\}$, class set $Y$,

    pre-trained GNN model $f_{\Theta_0}$ which takes in a graph and outputs its node embedding vectors.

    \Ensure Prompt vector $\vec{p}_t$.
    \State $\vec{p}_t\leftarrow$ prompt vector initialization, $\ensuremath{\mathcal{L}}_{\text{prompt}}\leftarrow 0$;
    \For{each graph $G_j\in \ensuremath{\mathcal{G}}$}\Comment{Load pre-trained GNN}
        \State $\vec{H}_j \leftarrow f_{\Theta_0}(G_j)$
    \EndFor
    \While{not converged} \Comment{Tuning iteration}
        \For{each subgraph $s_t,_x\in \ensuremath{\mathcal{S}}$}\Comment{Prompt design, Eq.~(12)}
            \State $\vec{s}_t,_x \leftarrow \textsc{ReadOut}(\{\vec{p}_t\odot\vec{h}_v:v\in V(S_x)\})$
        \EndFor
        \For {each class $c \in Y$}\Comment{Class prototypical subgraph}
            \State $\tilde{\vec{s}}_{t,c} \leftarrow $ Mean
            of node/graph embedding vectors

        \EndFor
        \For{each labeled pair $(x_i,y_i) \in \ensuremath{\mathcal{D}}$}\Comment{Accumulate loss, Eq.~(14)}
            \State $\vec{Z}_i \leftarrow 0$
            \For{each class $c \in Y$}
                \State $\vec{Z}_i=\exp(\text{sim}(\vec{s}_{t,x_i},\tilde{\vec{s}}_{t,c})/\tau)+\vec{Z}_i$
            \EndFor
            \State $\ensuremath{\mathcal{L}}_{\text{prompt}}=\ensuremath{\mathcal{L}}_{\text{prompt}}-\ln({\exp(\text{sim}(\vec{s}_{t,x_i},\tilde{\vec{s}}_{t,y_i})/\tau)}/{\vec{Z}_i})$
        \EndFor

        \State Update $\vec{p}_t$ by minimize $\ensuremath{\mathcal{L}}_{\text{prompt}}$;
    \EndWhile
    \State \Return $\vec{p}_t$.
\end{algorithmic}
\end{algorithm}

\stitle{Complexity analysis.}
For a node $v$, with average degree $\bar{d}$, $k$ GNN layers, $\delta$ hops for subgraph extraction, $D$ hidden dimensions, the complexity of GNN-based embedding calculation is $O(D\cdot \bar{d}^k)$, and the complexity of subgraph extraction is $O(\bar{d}^\delta)$. Thus, the embedding calculation of $v$'s subgraph with \textsc{ReadOut} is
$O(D\cdot \bar{d}^k \cdot \bar{d}^\delta)$, where $k,\delta$ are small constants.
Furthermore, if some neighborhood sampling \cite{hamilton2017inductive} is adopted during GNN aggregation,
$\bar{d}$ is a relatively small constant too.

\subsection{Further Descriptions of Datasets} \label{app.dataset}

We provide further details of the datasets.

(1) \emph{Flickr} \cite{wen2021meta} is an image sharing network,
which is collected by SNAP\footnote{\url{https://snap.stanford.edu/data/}}.
In particular, each node is an image, and there exists an edge between two images if they share some common properties, such as commented by the same user, or from the same location. Each image belongs to one of the 7 categories.

(2) \emph{PROTEINS} \cite{borgwardt2005protein} is a collection of protein graphs which include the amino acid sequence, conformation, structure, and features such as active sites of the proteins. The nodes represent the secondary structures, and each edge depicts the neighboring relation in the amino-acid sequence or in 3D space. The nodes belong to three categories, and the graphs belong to two classes.

(3) \emph{COX2} \cite{nr} is a dataset of molecular structures including 467 cyclooxygenase-2 inhibitors, in which each node is an atom, and each edge represents the chemical bond between atoms, such as single, double, triple or aromatic. All the molecules belong to two categories.

(4) \emph{ENZYMES} \cite{wang2022faith} is a dataset of 600 enzymes collected from the BRENDA enzyme database. These enzymes are labeled into 6 categories according to their top-level EC enzyme.

(5) \emph{BZR} \cite{nr} is a collection of 405 ligands for benzodiazepine receptor, in which each ligand is represented by a graph. All these ligands belong to 2 categories.

Note that we conduct node classification on \emph{Flickr}, \emph{PROTEINS} and \emph{ENZYMES}, since their node labels generally appear on all the graphs, which is suitable for the setting of few-shot node classification on each graph. Note that, we only choose the graphs which consist of more than 50 nodes for the downstream node classification, to ensure there exist sufficient labeled nodes for testing.
Additionally, graph classification is conducted on \emph{PROTEINS}, \emph{COX2}, \emph{ENXYMES} and \emph{BZR}. We use the given node features in the cited datasets to initialize input feature vectors, without additional processing.

\subsection{Further Descriptions of Baselines}

In this section, we present more details for the baselines, which are chosen from three main categories.

\vspace{1mm}
\noindent (1) \emph{End-to-end graph neural networks.}
\begin{itemize}[leftmargin=*]
    \item GCN \cite{kipf2016semi}: GCN resorts to mean-pooling based neighborhood aggregation to receive messages from the neighboring nodes for node representation learning in an end-to-end manner.
    \item GraphSage \cite{hamilton2017inductive}: GraphSAGE has a similar neighborhood aggregation mechanism with GCN, while it focuses more on the information from the node itself.
    \item GAT \cite{velivckovic2017graph} : GAT also depends on neighborhood aggregation for node representation learning in an end-to-end manner, while it can assign different weights to neighbors to reweigh their contributions.
    \item GIN \cite{xu2018powerful}: GIN employs a sum-based aggregator to replace the mean-pooling method in GCN, which is more powerful in expressing the graph structures.
\end{itemize}

\noindent (2) \emph{Graph pre-training models.}
\begin{itemize}[leftmargin=*]
    \item DGI \cite{velickovic2019deep}: DGI capitalizes on a self-supervised method for pre-training, which is based on the concept of mutual information (MI). It maximizes the  MI between the local augmented instances and the global representation.

    \item InfoGraph \cite{sun2019infograph}: InfoGraph learns a graph-level representation,
    which maximizes the MI  between the graph-level representation and substructure representations at various scales.
    \item GraphCL \cite{you2020graph}: GraphCL applies different graph augmentations to exploit the structural information on the graphs, and aims to maximize the agreement between different augmentations for graph pre-training.

\end{itemize}

\noindent (3) \emph{Graph prompt models.}
\begin{itemize}[leftmargin=*]
    \item GPPT \cite{sun2022gppt}. GPPT pre-trains a GNN model based on the link prediction task, and employs a learnable prompt to reformulate the downstream node classification task into the same format as link prediction.
\end{itemize}

\subsection{Further Implementation Details} \label{app.setting-parameter}

For baseline GCN \cite{kipf2016semi}, we employ a 3-layer architecture, and set the hidden dimension  as 32.
For GraphSAGE \cite{hamilton2017inductive}, we utilize the mean aggregator, and employ a 3-layer architecture. The hidden dimension is also set to 32.
For GAT \cite{velivckovic2017graph}, we employ a 2-layer architecture and set the hidden dimension as 32. Besides, we apply 4 attention heads in the first GAT layer.
Similarly, for GIN \cite{xu2018powerful}, we also employ a 3-layer architecture and set the hidden dimension as 32.
For the pre-training and prompting approaches, we use the backbones in their original paper.
Specifically, for DGI \cite{velickovic2019deep}, we use a 1-layer GCN as the backbone, and set the hidden dimension as 512. Besides, we utilize PReLU as the activation function.
For InfoGraph \cite{sun2019infograph}, we use a 3-layer GIN as the backbone, and set its hidden dimension as 32.
For GraphCL \cite{you2020graph}, we also employ a 3-layer GIN as its backbone, and set the hidden dimension as 32. In particular, we choose the augmentations of node dropping and subgraph, with a default augmentation ratio of 0.2.
For GPPT \cite{sun2022gppt}, we utilize a 2-layer GraphSAGE as its backbone, set its hidden dimension as 128, and utilize the mean aggregator.
For our proposed \method{GraphPrompt}{}, we employ a 3-layer GIN as the backbone, and set the hidden dimensions as 32. In addition, we set $\delta=1$ to construct 1-hop subgraphs for the nodes.

\subsection{Further Experimental Results} \label{app.experiments}

\stitle{Scalability study.}
We investigate the scalability of \method{GraphPrompt}{}\ on the dataset \emph{PROTEINS} for graph classification. We divide the graphs into six groups based on their size ({\it i.e.}, number of nodes). The size of graphs in each group is approximately 50, 60, $\ldots$, 100 nodes. We sample 10 graphs from each group, and record the prompt tuning time on the 10 graphs in each epoch. The results are presented in Fig.~\ref{fig.scalability}. Note that we also report the tuning time for \method{GraphPrompt}{}-ft, a variant of \method{GraphPrompt}{}, which fine-tunes all the parameters including the pre-trained GNN weights.
We first observe that the tuning time of our \method{GraphPrompt}{}\ increases linearly as the graph size increases, demonstrating the scalability of \method{GraphPrompt}{}\ on larger graphs. In addition, compared to \method{GraphPrompt}{}, \method{GraphPrompt}{}-ft needs more tuning time, showing the inefficiency of the fine-tuning paradigm.

\begin{figure}[tbh]
\centering
\includegraphics[width=0.55\linewidth]{figures/scalability-tune-www.pdf}
\vspace{-2mm}
\caption{Scalability study.}
\label{fig.scalability}
\end{figure}

\stitle{Parameter sensitivity.}
We evaluate the sensitivity of two important hyperparameters in \method{GraphPrompt}{}, and show the impact in Figs.~\ref{fig.tune-node} and \ref{fig.tune-graph} for node classification and graph classification, respectively.

For the number of hops ($\delta$) in subgraph construction, the performance on node classification gradually decreases as the number of hops increases. This is because a larger subgraph tends to bring in irrelevant information for the target node, and may suffer from the over-smoothing issue \cite{chen2020measuring}. On the other hand, for graph classification, the number of hops only affects the pre-training stage as the whole graph is used in downstream classification. In this case, the number of hops does not show a clear trend, implying less impact on graph classification since both small and large subgraphs are helpful in capturing  substructure information at different scales.

For the hidden dimension, a smaller dimension is better for node classification, such as 32 and 64. For graph classification, a slightly larger dimension might be better, such as 64 and 128. Overall, 32 or 64 appears to be robust  for both node and graph classification.

\begin{figure}[htb]
  \centering
  \subfigure[Number of hops]{\label{fig.node-params-tune-K}\includegraphics[width=0.48\linewidth]{figures/node-params-tune-K.pdf}}
  \subfigure[Hidden dimension]{\label{fig.node-params-tune-d}\includegraphics[width=0.48\linewidth]{figures/node-params-tune-d.pdf}}
     \vspace{-2mm}
  \caption{Parameter sensitivity on node classification.}
\label{fig.tune-node}
\end{figure}

\begin{figure}[htb]
  \centering
  \subfigure[Number of hops]{\label{fig.graph-params-tune-K}\includegraphics[width=0.48\linewidth]{figures/graph-params-tune-K.pdf}}
  \subfigure[Hidden dimension]{\label{fig.graph-params-tune-d}\includegraphics[width=0.48\linewidth]{figures/graph-params-tune-d.pdf}}
     \vspace{-2mm}
  \caption{Parameter sensitivity on graph classification.}
\label{fig.tune-graph}
\end{figure}

\subsection{Data Ethics Statement}
To evaluate the efficacy of this
work, we conducted experiments which only use publicly available
datasets, namely, Flickr\footnote{\url{https://snap.stanford.edu/data/web-flickr.html}}, PROTEINS, COX2, ENZYMES and BZR\footnote{\url{https://chrsmrrs.github.io/datasets/}}, in accordance to their usage terms
and conditions if any.
We further declare that no personally identifiable information was
used, and no human or animal subject was involved in this research.

\section{Conclusions}

In this paper, we studied the research problem of prompting on graphs and proposed \method{GraphPrompt}{}, in order to overcome the limitations of graph neural networks in the supervised or ``pre-train, fine-tune'' paradigms.
In particular, to narrow the gap between pre-training and downstream objectives on graphs, we introduced a unification framework by mapping different tasks to a common task template. Moreover, to distinguish task individuality and achieve  task-specific optima, we proposed a learnable task-specific prompt vector that guides each downstream task to make full of the pre-trained model.
Finally, we conduct extensive experiments on five public datasets, and show that \method{GraphPrompt}{}\ significantly outperforms various state-of-the-art baselines.

\section*{Acknowledgments}
This research / project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (MOE-T2EP20122-0041). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.
\section{Experiments}
In this section, we conduct extensive experiments including node classification and graph classification as downstream tasks on five benchmark datasets to evaluate the proposed \method{GraphPrompt}{}.

\subsection{Experimental Setup}

\stitle{Datasets.}
We employ five benchmark datasets for evaluation.
(1) \emph{Flickr} \cite{wen2021meta} is an image sharing network.
(2) \emph{PROTEINS} \cite{borgwardt2005protein} is a collection of protein graphs which include the amino acid sequence, conformation, structure, and features such as active sites of the proteins.
(3) \emph{COX2} \cite{nr} is a dataset of molecular structures including 467 cyclooxygenase-2 inhibitors.
(4) \emph{ENZYMES} \cite{wang2022faith} is a dataset of 600 enzymes collected from the BRENDA enzyme database.
(5) \emph{BZR} \cite{nr} is a collection of 405 ligands for benzodiazepine receptor.

We summarize these datasets in Table~\ref{table.datasets}, and present further details in Appendix~\ref{app.dataset}.
Note that the ``Task'' column indicates the type of downstream task performed on each dataset: ``N'' for node classification and ``G'' for graph classification.

\begin{table}[tbp]
\center
\addtolength{\tabcolsep}{-0.5mm}
\caption{Summary of datasets.
\label{table.datasets}}
\vspace{-2mm}
\small
\begin{tabular}{@{}c|rrrrrrc@{}}
\toprule
	& \makecell[c]{Graphs} & \makecell[c]{Graph \\ classes} & \makecell[c]{Avg.\\ nodes} & \makecell[c]{Avg. \\ edges} &  \makecell[c]{Node \\ features} &  \makecell[c]{Node \\ classes} & \makecell[c]{Task \\ (N/G)} \\
\midrule
     Flickr & 1 & - & 89,250 & 899,756 & 500 & 7 & N \\
     PROTEINS & 1,113 & 2 & 39.06 & 72.82 & 1 & 3 & N, G\\
     COX2 & 467 & 2 & 41.22 & 43.45 & 3 & - & G\\
     ENZYMES & 600 & 6 & 32.63 & 62.14 & 18 & 3 & N, G\\
     BZR & 405 & 2 & 35.75 & 38.36 & 3 & - & G\\
 \bottomrule
\end{tabular
\end{table}

\stitle{Baselines.}
We evaluate \method{GraphPrompt}{}\ against the state-of-the-art approaches from three main categories, as follows.
(1) \emph{End-to-end graph neural networks}: GCN \cite{kipf2016semi}, GraphSAGE \cite{hamilton2017inductive}, GAT \cite{velivckovic2017graph} and GIN \cite{xu2018powerful}. They capitalize on the key operation of neighborhood aggregation to recursively aggregate messages from the neighbors, and work in an end-to-end manner.
(2) \emph{Graph pre-training models}: DGI \cite{velickovic2019deep}, InfoGraph \cite{sun2019infograph}, and GraphCL \cite{you2020graph}. They work in the ``pre-train, fine-tune'' paradigm. In particular, they pre-train the GNN models to preserve the intrinsic graph properties, and fine-tune the pre-trained weights on downstream tasks to fit task labels.
(3) \emph{Graph prompt models}: GPPT \cite{sun2022gppt}. GPPT utilizes a link prediction task for pre-training, and resorts to a learnable prompt for the node classification task, which is mapped to a link prediction task.

Note that other few-shot learning methods on graphs, such as Meta-GNN \cite{zhou2019meta} and RALE \cite{liu2021relative}, adopt a meta-learning paradigm \cite{finn2017model}. Thus, they cannot be used in our setting, as they require labeled data in their base classes for the meta-training phase. In our approach, only label-free graphs are utilized for pre-training.

\stitle{Settings and parameters.}
To evaluate the goal of our \method{GraphPrompt}{}\ in realizing a unified design that can suit different downstream tasks flexibly, we consider two typical types of downstream tasks, {\it i.e.}, node classification and graph classification.
In particular, for the datasets which are suitable for both of these two tasks, {\it i.e.}, \emph{PROTEINS} and \emph{ENZYMES}, we only pre-train the GNN model once on each dataset, and utilize the same pre-trained model for the two downstream tasks with their task-specific prompting.

The downstream tasks follow a $k$-shot classification setting. For each type of downstream task, we construct a series of $k$-shot classification tasks. The details of task construction will be elaborated later when reporting the results in Sect.~\ref{sec:expt:perf}.
For task evaluation, as the $k$-shot tasks are balanced classification, we employ accuracy as the evaluation metric following earlier work \cite{wang2020graph,liu2021relative}.

For all the baselines, based on the authors' code and default settings, we further tune their hyper-parameters to optimize their performance. We present more implementation details of the baselines and our \method{GraphPrompt}{}\ in Appendix~\ref{app.setting-parameter}.

\subsection{Performance Evaluation}\label{sec:expt:perf}

As discussed, we perform two types of downstream task different from the link prediction task in pre-training, namely, node classification and graph classification in few-shot settings. We first evaluate on a fixed-shot setting, and then vary the shot numbers to see the performance trend.

\stitle{Few-shot node classification.}
We conduct this node-level task on three datasets, {\it i.e.}, \emph{Flickr}, \emph{PROTEINS}, and \emph{ENZYMES}. Following a typical $k$-shot setup \cite{zhou2019meta,wang2020graph,liu2021relative}, we generate a series of few-shot tasks for model training and validation.
In particular, for \emph{PROTEINS} and \emph{ENZYMES}, on each graph we randomly generate ten 1-shot node classification tasks ({\it i.e.}, in each task, we randomly sample 1 node per class) for training and validation, respectively. Each training task is paired with a validation task, and the remaining nodes not sampled by the pair of training and validation tasks will be used for testing. For \emph{Flickr}, as it contains a large number of very sparse node features, selecting very few shots for training may result in inferior performance for all the methods.
Therefore, we randomly generate ten 50-shot node classifcation tasks, for training and validation, respectively. On Flickr, 50 shots are still considered few, accounting for less than 0.06\% of all nodes on the graph.

Table~\ref{table.node-classification} illustrates the results of few-shot node classification. We have the following observations.
First, our proposed \method{GraphPrompt}{}\ outperforms all the baselines across the three datasets, demonstrating the effectiveness of \method{GraphPrompt}{}\ in transferring knowledge from the pre-training to downstream tasks.
In particular, by virtue of the unification framework and prompt-based task-specific aggregation in $\textsc{ReadOut}$ function, \method{GraphPrompt}{}\ is able to narrow the gap between pre-training and downstream tasks, and guide the downstream tasks to exploit the pre-trained model in a task-specific manner.
Second, compared to graph pre-training models, end-to-end GNN models can sometimes achieve comparable or even better performance. This implies that the discrepancy between the pre-training and downstream tasks in these pre-training approaches obstructs the knowledge transfer from the former to the latter. In such a case, even with sophisticated pre-training, they cannot effectively promote the performance of downstream tasks.
Third, the graph prompt model GPPT is only comparable to or even worse than the other baselines, despite also using prompts. A potential reason is that GPPT requires much more learnable parameters in their prompts than ours, which may not work well given very few shots ({\it e.g.}, 1-shot).

\stitle{Few-shot graph classification.}
We further conduct few-shot graph classification on four datasets, {\it i.e.}, \emph{PROTEINS}, \emph{COX2}, \emph{ENZYMES}, and \emph{BZR}.
For each dataset, we randomly generate 100 5-shot classification tasks for training and validation, following a similar process for node classification tasks.

We illustrate the results of few-shot graph classification in Table~\ref{table.graph-classification}, and have the following observations.
First, our proposed \method{GraphPrompt}{}\ significantly outperforms the baselines on these four datasets. This again demonstrates the necessity of unification for pre-training and downstream tasks, and the effectiveness of prompt-assisted task-specific aggregation for $\textsc{ReadOut}$.
Second, as both node and graph classification tasks share the same pre-trained model on \emph{PROTEINS} and \emph{ENZYMES}, the superior performance of \method{GraphPrompt}{}\ on both types of task further demonstrates that, the gap between different tasks
is well addressed by virtue of our unification framework.
Third, the graph pre-training models generally achieve better performance than the end-to-end GNN models.
This is because both InfoGraph and GraphCL capitalize on graph-level tasks for pre-training, which are naturally closer to the downstream graph classification.

\begin{table}[tbp]
    \centering
    \small

     \addtolength{\tabcolsep}{1mm}
    \caption{Accuracy evaluation on node classification.

    }
    \label{table.node-classification}%
    \vspace{-3mm}
    {\footnotesize All tabular results are in percent, with best \textbf{bolded} and runner-up \underline{underlined}.}
    \\[1mm]

    \begin{tabular}{@{}l|c|c|c@{}}
    \toprule
  \multirow{2}*{Methods} & \multicolumn{1}{c|}{Flickr} & \multicolumn{1}{c|}{PROTEINS} & \multicolumn{1}{c}{ENZYMES}  \\ & 50-shot & 1-shot & 1-shot  \\\midrule\midrule
    {GCN} &  9.22 $\pm$ 9.49 & 59.60 $\pm$ 12.44 & 61.49 $\pm$ 12.87 \\
    {GraphSAGE}  & 13.52 $\pm$ 11.28 & 59.12 $\pm$ 12.14 & 61.81 $\pm$ 13.19\\
     {GAT} & 16.02 $\pm$ 12.72 & 58.14 $\pm$ 12.05 & 60.77 $\pm$ 13.21\\
    {GIN} & 10.18 $\pm$ 5.41 & \underline{60.53} $\pm$ 12.19  & \underline{63.81} $\pm$ 11.28
\\\midrule
    {DGI} & 17.71 $\pm$ 1.09 & 54.92 $\pm$ 18.46 & 63.33 $\pm$ 18.13\\
    {GraphCL}  & 18.37 $\pm$ 1.72 & 52.00 $\pm$ 15.83 & 58.73 $\pm$ 16.47\\
    \midrule
    {GPPT} & \underline{18.95} $\pm$ 1.92 & 50.83 $\pm$ 16.56 & 53.79 $\pm$ 17.46\\
    \midrule
    {GraphPrompt} & \textbf{20.21} $\pm$ 11.52 & \textbf{63.03} $\pm$ 12.14 & \textbf{67.04} $\pm$ 11.48\\
    \bottomrule
        \end{tabular

\end{table}

\begin{table}[tbp]
    \centering
    \small

     \addtolength{\tabcolsep}{-1mm}
    \caption{Accuracy evaluation on graph classification.}
    \vspace{-2mm}
    \label{table.graph-classification}%

    \begin{tabular}{@{}l|c|c|c|c@{}}
    \toprule
  \multirow{2}*{Methods} & \multicolumn{1}{c|}{PROTEINS} & \multicolumn{1}{c|}{COX2} & \multicolumn{1}{c|}{ENZYMES} & \multicolumn{1}{c}{BZR}  \\ & 5-shot  & 5-shot & 5-shot & 5-shot \\\midrule\midrule

    {GCN} & 54.87 $\pm$ 11.20 & 51.37 $\pm$ 11.06  & 20.37 $\pm$ 5.24	 & 56.16 $\pm$ 11.07	\\
    {GraphSAGE} & 52.99 $\pm$ 10.57 & 52.87 $\pm$ 11.46	 & 18.31 $\pm$ 6.22 & 57.23 $\pm$ 10.95\\
     {GAT} & 48.78 $\pm$ 18.46 & 51.20 $\pm$ 27.93 & 15.90 $\pm$ 4.13 & 53.19 $\pm$ 20.61\\
    {GIN} & \underline{58.17} $\pm$ 8.58 & 51.89 $\pm$ 8.71	& 20.34 $\pm$ 5.01 & 57.45 $\pm$ 10.54\\
    \midrule
    {InfoGraph} & 54.12 $\pm$ 8.20 & 54.04 $\pm$ 9.45 & 20.90 $\pm$ 3.32 & 57.57 $\pm$ 9.93\\
    {GraphCL} & 56.38 $\pm$ 7.24  & \underline{55.40} $\pm$ 12.04  & \underline{28.11} $\pm$ 4.00  & \underline{59.22} $\pm$ 7.42	\\
    \midrule
    {GraphPrompt} & \textbf{64.42} $\pm$ 4.37  & \textbf{59.21} $\pm$ 6.82 & \textbf{31.45} $\pm$ 4.32	 & \textbf{61.63} $\pm$ 7.68 \\\bottomrule
    \end{tabular

\end{table}

\stitle{Performance with different shots.}
We study the impact of number of shots on the \emph{PROTEINS} and \emph{ENZYMES} datasets. For node classification, we vary the number of shots between 1 and 10, and compare with several competitive baselines ({\it i.e.}, GIN, DGI, GraphCL, and GPPT) in Fig.~\ref{fig.node-few-shot-tune}.
For few-shot graph classification, we vary the number of shots between 1 and 30, and compare with competitive baselines ({\it i.e.}, GIN, InfoGraph, and GraphCL) in Fig.~\ref{fig.graph-few-shot-tune}.
The task settings are identical to those stated earlier.

In general, our proposed \method{GraphPrompt}{}\ consistently outperforms the baselines especially with lower shots. For node classification, as the number of nodes in each graph is relatively small, 10 shots per class might be sufficient for semi-supervised node classification. Nevertheless, \method{GraphPrompt}{}\ is competitive even with 10 shots. For graph classification, \method{GraphPrompt}{}\ can be surpassed by some baselines when given more shots ({\it e.g.}, 20 or more), especially on \emph{ENZYMES}.
On this dataset, 30 shots per class implies 30\% of the 600 graphs are used for training, which is not our target scenario.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/node-few-shot-tune.pdf}
\vspace{-3mm}
\caption{Impact of shots on few-shot node classification.}
\label{fig.node-few-shot-tune}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/graph-few-shot-tune.pdf}
\vspace{-3mm}
\caption{Impact of shots on few-shot graph classification.}
\label{fig.graph-few-shot-tune}
\end{figure}

\subsection{Model Analysis}
We further analyse several aspects of our model. Due to space constraint, we only report the ablation and parameter efficiency study, and leave the rest to Appendix~\ref{app.experiments}.

\begin{figure}[t]
  \centering
  \subfigure[Node classification]{\label{fig.node-ablation}\includegraphics[width=0.48\linewidth]{figures/node-ablation-acc.pdf}}
  \subfigure[Graph classification]{\label{fig.graph-ablation}\includegraphics[width=0.48\linewidth]{figures/graph-ablation-acc.pdf}}
     \vspace{-3mm}
  \caption{Ablation study.}
\label{fig.ablation}
\end{figure}

\stitle{Ablation study.}
To evaluate the contribution of each component, we conduct an ablation study by comparing \method{GraphPrompt}{}\ with different prompting strategies:
(1) \emph{no prompt}: for downstream tasks, we remove the prompt vector, and conduct classification by employing a classifier on the subgraph representations obtained by a direct sum-based $\textsc{ReadOut}$.
(2) \emph{lin. prompt}: we replace the prompt vector with a linear transformation matrix in Eq.~\eqref{eq:prompt-lin}.

We conduct the ablation study on three datasets for node classification (\emph{Flickr}, \emph{PROTEINS}, and \emph{ENZYMES}) and graph classification (\emph{COX2}, \emph{ENZYMES}, and \emph{BZR}), respectively, and illustrate the comparison in Fig.~\ref{fig.ablation}. We have the following observations.
(1) Without the prompt vector, \emph{no prompt} usually performs the worst among the variants, showing the necessity of prompting the $\textsc{ReadOut}$ operation differently for different downstream tasks.
(2) Converting the prompt vector into a linear transformation matrix also hurts the performance, as the matrix involves more parameters thus increasing the reliance on labeled data.

\stitle{Parameter efficiency.}
We also compare the number of parameters that needs to be updated in a downstream node classification task  for a few representative models, as well as their number of floating point operations (FLOPs), in Table~\ref{table.parameters-num}.

In particular, as GIN works in an end-to-end manner, it is obvious that it involves the largest number of parameters for updating.
For GPPT, it requires a separate learnable vector for each class as its representation, and an attention module to weigh the neighbors for aggregation in the structure token generation. Therefore, GPPT needs to update more parameters than \method{GraphPrompt}{}, which is one factor that impairs its performance in downstream tasks.
For our proposed \method{GraphPrompt}{}, it not only outperforms the baselines GIN and GPPT as we have seen earlier, but also requires the least parameters and FLOPs for downstream tasks. For illustration, in addition to prompt tuning,
if we also fine-tune the pre-trained weights instead of freezing them (denoted \method{GraphPrompt}{}+ft), there will be significantly more parameters to update.

\begin{table}[!t]
    \centering
    \small
    \addtolength{\tabcolsep}{-1.5pt}

   \caption{Study of parameter efficiency on node classification.}

    \begin{tabular}{@{}l|cc|cc|cc@{}}
    \toprule
  \multirow{2}*{Methods} & \multicolumn{2}{c|}{Flickr} & \multicolumn{2}{c|}{PROTEINS} & \multicolumn{2}{c}{ENZYMES}\\
  & Params & FLOPs & Params & FLOPs & Params & FLOPs \\\midrule\midrule
    GIN & 22,183 &240,100 & 5,730 & 12,380 & 6,280 & 11,030\\
    GPPT & 4,096 & 4,582 & 1,536 & 1,659 & 1,536 & 1,659 \\ \midrule

    \method{GraphPrompt}{}\ & 96 &96 & 96 &96 & 96 &96   \\
    \method{GraphPrompt}{}-ft &21,600 &235,200 &6,176 &13,440 &6,176 &10,944\\
\bottomrule
     \end{tabular
      \label{table.parameters-num}%

\end{table}

\section{Introduction}

The ubiquitous Web is becoming the ultimate data repository, capable of linking a broad spectrum of objects to form gigantic and complex graphs. The prevalence of graph data enables a series of downstream tasks for Web applications, ranging from online page/article classification to friend recommendation in social networks.
Modern approaches for graph analysis generally resort to graph representation learning including graph embedding and graph neural networks (GNNs). Earlier graph embedding approaches \cite{perozzi2014deepwalk,tang2015line,grover2016node2vec} usually embed nodes on the graph into a low-dimensional space, in which the structural information such as the proximity between nodes can be captured \cite{cai2018comprehensive}.
More recently, GNNs \cite{kipf2016semi,hamilton2017inductive,velivckovic2017graph,xu2018powerful} have emerged as the state of the art for graph representation learning. Their key idea boils down to a message-passing framework, in which each node derives its representation by receiving and aggregating messages from its neighboring nodes recursively \cite{wu2020comprehensive}.

\stitle{Graph pre-training.}
Typically, GNNs work in an end-to-end manner, and their performance depends heavily on the availability of large-scale, task-specific labeled data as supervision. This supervised paradigm presents two problems. First, task-specific supervision is often difficult or costly to obtain. Second, to deal with a new task, the weights of GNN models need to be retrained from scratch, even if the task is on the same graph.
To address these issues, pre-training GNNs \cite{hu2020gpt,Hu2020Strategies,qiu2020gcc,lu2021learning} has become increasingly popular, inspired by pre-training techniques in language and vision applications \cite{dong2019unified,bao2022beit}.
The pre-training of GNNs leverages self-supervised learning on more readily available label-free graphs ({\it i.e.}, graphs without task-specific labels), and learns intrinsic graph properties that intend to be general across tasks and graphs in a domain. In other words, the pre-training extracts a task-agnostic prior, and can be used to initialize model weights for a new task. Subsequently, the initial weights can be quickly updated through a lightweight fine-tuning step on a smaller number of task-specific labels.

However, the ``pre-train, fine-tune'' paradigm suffers from the problem of inconsistent objectives between  pre-training and downstream tasks, resulting in suboptimal performance \cite{liu2021pre}.
On one hand, the pre-training step aims to preserve various intrinsic graph properties such as node/edge features \cite{Hu2020Strategies,hu2020gpt}, node connectivity/links \cite{hamilton2017inductive,hu2020gpt,lu2021learning}, and local/global patterns \cite{qiu2020gcc,Hu2020Strategies,lu2021learning}.
On the other hand, the fine-tuning step aims to reduce the task loss, {\it i.e.}, to fit the ground truth of the downstream task.
The discrepancy between the two steps can be quite large.
For example, pre-training may focus on learning the connectivity pattern between two nodes ({\it i.e.}, related to link prediction), whereas fine-tuning could be dealing with a node or graph property  ({\it i.e.}, node classification or graph classification task).

\stitle{Prior work.} To narrow the gap between pre-training and downstream tasks, prompting \cite{brown2020language} has first been proposed for language models, which is a natural language instruction designed for a specific downstream task to ``prompt out'' the semantic relevance between the task and the language model. Meanwhile, the parameters of the pre-trained language model are frozen without any fine-tuning, as the prompt can ``pull'' the task toward the pre-trained model. Thus, prompting is also more efficient than fine-tuning, especially when the pre-trained model is huge.
Recently, prompting has also been introduced to graph pre-training in the GPPT approach  \cite{sun2022gppt}.
While the pioneering work has proposed a sophisticated design of pre-training and prompting, it can only be employed for the node classification task, lacking a universal treatment that  appeals to different downstream tasks  such as both node classification and graph classification.

\begin{figure}[t]
\centering
\includegraphics[width=0.99\linewidth]{figures/intro.pdf}
\caption{Illustration of the motivation. (a) Pre-training on graphs. (b/c) Downstream node/graph classification.}
\label{fig.intro-motivation}
\end{figure}

\stitle{Research problem and challenges.}
To address the divergence between graph pre-training and various downstream tasks, in this paper we investigate the design of pre-training and prompting for graph neural networks. In particular, we aim for a unified design that can suit different downstream tasks flexibly.
This problem is non-trivial due to the following two challenges.

Firstly, to enable effective knowledge transfer from the pre-training to a downstream task, it is desirable that the pre-training step preserves graph properties that are compatible with the given task. However, since different downstream tasks often have different objectives, \emph{how do we unify pre-training with various downstream tasks on graphs}, so that a single pre-trained model can universally support different tasks? That is, we try to convert the pre-training task and downstream tasks to follow the same ``template''. Using pre-trained language models as an analogy, both their pre-training and downstream tasks can be formulated as masked language modeling.

Secondly, under the unification framework, it is still important to identify the distinction between different downstream tasks, in order to attain task-specific optima. For pre-trained language models, prompts in the form of natural language tokens or learnable word vectors have been designed to give different hints to different tasks, but it is less apparent what form prompts on graphs should take.
Hence, \emph{how do we design prompts on graphs}, so that they can guide different downstream tasks to effectively make use of the pre-trained model?

\stitle{Present work.}
To address these challenges, we propose a novel graph pre-training and prompting framework, called  \method{GraphPrompt}{}, aiming to unify the pre-training and downstream tasks for GNNs.
Drawing inspiration from the prompting strategy for pre-trained language models, \method{GraphPrompt}{}\ capitalizes on a unified template to define the objectives for both pre-training and downstream tasks, thus bridging their gap. We further equip \method{GraphPrompt}{}\ with task-specific learnable prompts, which guides the downstream task to exploit relevant knowledge from the pre-trained GNN model.
The unified approach endows \method{GraphPrompt}{}\ with the ability of working on limited supervision such as few-shot learning tasks.

More specifically, to address the first challenge of unification, we focus on graph topology, which is a key enabler of graph models. In particular, subgraph is a universal structure that can be leveraged for both node- and graph-level tasks. At the node level, the information of a node can be enriched and represented by its contextual subgraph, {\it i.e.}, a subgraph where the node resides in \cite{zhang2018link,huang2020graph}; at the graph level, the information of a graph is naturally represented by the maximum subgraph ({\it i.e.}, the graph itself).
Consequently, we unify both the node- and graph-level tasks, whether in pre-training or downstream, into the same template: the similarity calculation of (sub)graph\footnote{As a graph is a subgraph of itself, we may simply use subgraph to refer to a graph too.} representations.
In this work, we adopt link prediction as the self-supervised pre-training task, given that links are readily available in any graph without additional annotation cost. Meanwhile, we focus on the popular node classification and graph classification as downstream tasks, which are node- and graph-level tasks, respectively. All these tasks can be cast as instances of learning subgraph similarity. On one hand, the link prediction task in pre-training boils down to the similarity  between the contextual subgraphs of two nodes, as shown in Fig.~\ref{fig.intro-motivation}(a). On the other hand, the downstream node or graph classification task boils down to the similarity between the target instance (a node's contextual subgraph or the whole graph, resp.) and the class prototypical subgraphs constructed from labeled data, as illustrated in Figs.~\ref{fig.intro-motivation}(b) and (c). The unified template bridges the gap between the pre-training and different downstream tasks.

Toward the second challenge, we distinguish different downstream tasks by way of the $\textsc{ReadOut}$ operation on subgraphs. The $\textsc{ReadOut}$ operation is essentially an aggregation function to fuse node representations in the subgraph into a single subgraph representation. For instance, sum pooling, which sums the representations of all nodes in the subgraph, is a practical and popular scheme for $\textsc{ReadOut}$. However, different downstream tasks can benefit from different aggregation schemes for their $\textsc{ReadOut}$. In particular, node classification tends to focus on features that can contribute to the representation of the target node, while  graph classification tends to focus on features associated with the graph class.
Motivated by such differences, we propose a novel task-specific learnable prompt to guide the $\textsc{ReadOut}$ operation of each downstream task with an appropriate aggregation scheme.
As shown in Fig.~\ref{fig.intro-motivation}, the learnable prompt serves as the parameters of the $\textsc{ReadOut}$ operation of downstream tasks, and thus enables different aggregation functions on the subgraphs of different tasks.
Hence, \method{GraphPrompt}{}\ not only unifies the pre-training and downstream tasks into the same template based on subgraph similarity, but also recognizes the differences between various downstream tasks to guide task-specific objectives.

\stitle{Contributions.}
To summarize, our contributions are three-fold.
(1) We recognize the gap between graph pre-training and downstream tasks, and propose a unification framework \method{GraphPrompt}{}\ based on subgraph similarity for both pre-training and downstream tasks, including both node and graph classification tasks.
(2) We propose a novel prompting strategy for \method{GraphPrompt}{}, hinging on a learnable prompt to actively guide downstream tasks using task-specific aggregation in $\textsc{ReadOut}$, in order to drive the downstream tasks to exploit the pre-trained model in a task-specific manner.
(3) We conduct extensive experiments on five public datasets, and the results demonstrate the superior performance of \method{GraphPrompt}{}\ in comparison to the state-of-the-art approaches.

\section{Proposed Approach}

In this section, we present our proposed approach \method{GraphPrompt}{}.

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{figures/framework.pdf}
\vspace{-6mm}
\caption{Overall framework of \method{GraphPrompt}{}.}
\label{fig.framework}
\end{figure*}

\subsection{Unification Framework}\label{sec:model:unification}

We first introduce the overall  framework of \method{GraphPrompt}{}\ in Fig.~\ref{fig.framework}.
Our framework is deployed on a set of label-free graphs shown in  Fig.~\ref{fig.framework}(a), for pre-training in  Fig.~\ref{fig.framework}(b).
The pre-training adopts a link prediction task, which is self-supervised without requiring extra annotation.
Afterward, in Fig.~\ref{fig.framework}(c), we capitalize on a learnable prompt to guide each downstream task, namely, node classification or graph classification, for task-specific exploitation of the pre-trained model.
In the following, we explain how the framework supports a unified view of pre-training and downstream tasks.

\stitle{Instances as subgraphs.} The key to the unification of pre-training and downstream tasks lies in finding a common template for the tasks. The task-specific prompt can then be further fused with the template of each downstream task, to distinguish the varying characteristics of different tasks.

In comparison to other fields such as visual and language processing,
graph learning is uniquely characterized by the exploitation of graph topology.
In particular, subgraph is a universal structure capable of expressing both node- and graph-level instances.
On one hand, at the node level, every node resides in a local neighborhood, which in turn contextualizes the node \cite{DBLP:conf/cikm/LiuZFZH20,liu2021nodewise,liu2021tail}. The local neighborhood of a node $v$ on a graph $G=(V,E)$ is usually defined by a contextual subgraph $S_v=(V(S_v), E(S_v))$, where its set of nodes and edges are respectively given by
\begin{align}
V(S_v)&=\{d(u,v)\le \delta \mid u\in V\}, \text{ and }\\
E(S_v)&=\{ (u,u')\in E \mid u \in V(S_v), u'\in V(S_v) \},
\end{align}
where $d(u,v)$ gives the shortest distance between nodes $u$ and $v$ on the graph $G$, and $\delta$ is a predetermined threshold. That is, $S_v$ consists of nodes within $\delta$ hops from the node $v$, and the edges between those nodes. Thus, the contextual subgraph $S_v$ embodies not only the self-information of the node $v$, but also rich contextual information to complement the self-information \cite{zhang2018link,huang2020graph}.
On the other hand, at the graph level, the maximum subgraph of a graph $G$, denoted $S_{G}$, is the graph itself, {\it i.e.}, $S_{G}=G$. The maximum subgraph  $S_{G}$ spontaneously embodies all information of $G$.
In summary, subgraphs can be used to represent both node- and graph-level instances: Given an instance $x$ which can either be a node or a graph ({\it e.g.}, $x=v$ or $x=G$), the subgraph $S_x$ offers a unified access to the information associated with $x$.

\stitle{Unified task template.} Based on the above subgraph definitions for both node- and graph-level instances, we are ready to unify different tasks to follow a common template. Specifically, the link prediction task in pre-training and the downstream node and graph classification tasks can all be redefined as \emph{subgraph similarity learning}. Let $\vec{s}_x$ be the vector representation of the subgraph $S_x$, and $\text{sim}(\cdot,\cdot)$ be the cosine similarity function. As illustrated in Figs.~\ref{fig.framework}(b) and (c), the three tasks can be mapped to the computation  of subgraph similarity, which is formalized below.
\begin{itemize}[leftmargin=*]
    \item \textbf{\textit{Link prediction}}: This is a node-level task. Given a graph $G=(V,E)$ and a triplet of nodes $(v,a,b)$ such that $(v,a)\in E$ and $(v,b)\notin E$, we shall have
    \begin{align}
    \text{sim}(\vec{s}_v,\vec{s}_a)>\text{sim}(\vec{s}_v,\vec{s}_b).
    \end{align}
    Intuitively, the contextual subgraph of $v$ shall be more similar to that of a node linked to $v$ than that of another unlinked node.

    \item \textbf{\textit{Node classification}}: This is also a node-level task. Consider a graph $G=(V,E)$ with a set of  node classes $C$, and a set of labeled nodes $D=\{(v_1,\ell_1),(v_2,\ell_2),\ldots\}$ where $v_i \in V$ and  $\ell_i$ is the corresponding label of $v_i$. As we adopt a $k$-shot setting, there are exactly $k$ pairs of $(v_i,\ell_i=c) \in D$ for every class $c\in C$.
    For each class $c \in C$, further define a \emph{node class prototypical subgraph} represented by a vector $\vec{\tilde{s}}_c$, given by
    \begin{align}
        \vec{\tilde{s}}_c = \frac{1}{k}\sum_{(v_i,\ell_i)\in D, \ell_i=c} \vec{s}_{v_i}.
    \end{align}
    Note that the class prototypical subgraph is a ``virtual'' subgraph with a latent representation in the same embedding space as the node contextual subgraphs. Basically, it is constructed as the mean representation of the contextual subgraphs of labeled nodes in a given class.
    Then, given a node $v_j$ not in the labeled set $D$, its class label $\ell_j$ shall be
    \begin{align}
        \ell_j = \arg\max_{c\in C} \text{sim}(\vec{s}_{v_j}, \vec{\tilde{s}}_c).
    \end{align}
    Intuitively, a node shall belong to the class whose prototypical subgraph is the most similar to the node's contextual subgraph.

    \item \textbf{\textit{Graph classification}}: This is a graph-level task. Consider a set of graphs $\ensuremath{\mathcal{G}}$ with a set of graph classes $\ensuremath{\mathcal{C}}$, and a set of labeled graphs $\ensuremath{\mathcal{D}}=\{(G_1,L_1),(G_2,L_2),\ldots\}$ where $G_i \in \ensuremath{\mathcal{G}}$ and  $L_i$ is the corresponding label of $G_i$. In the $k$-shot setting, there are exactly $k$ pairs of $(G_i,L_i=c) \in \ensuremath{\mathcal{D}}$ for every class $c\in \ensuremath{\mathcal{C}}$.
    Similar to node classification, for each class $c \in \ensuremath{\mathcal{C}}$, we define a \emph{graph class prototypical subgraph}, also represented by the mean embedding vector of the (sub)graphs in $c$:
    \begin{align}
        \vec{\tilde{s}}_c = \frac{1}{k}\sum_{(G_i,L_i)\in \ensuremath{\mathcal{D}}, L_i=c} \vec{s}_{G_i}.
    \end{align}
    Then, given a graph $G_j$ not in the labeled set $\ensuremath{\mathcal{D}}$, its class label $L_j$ shall be
    \begin{align}
        L_j = \arg\max_{c\in \ensuremath{\mathcal{C}}} \text{sim}(\vec{s}_{G_j}, \vec{\tilde{s}}_c).
    \end{align}
    Intuitively, a graph shall belong to the class whose prototypical subgraph is the most similar to itself. \hspace*{\fill}\qed
\end{itemize}

It is worth noting that node and graph classification can be further condensed into a single set of notations. Let $(x,y)$ be an annotated instance of graph data, {\it i.e.}, $x$ is either a node or a graph, and $y\in Y$ is the class label of $x$ among the set of classes $Y$. Then,
\begin{align}
        y = \arg\max_{c\in Y} \text{sim}(\vec{s}_{x}, \vec{\tilde{s}}_c).
    \end{align}

Finally, to materialize the common task template, we discuss how to learn the subgraph embedding vector $\vec{s}_x$ for the subgraph $S_x$.
Given node representations $\vec{h}_{v}$ generated by a GNN (see Sect.~\ref{sec:prelim:gnn}), a standard approach of computing $\vec{s}_x$ is to employ a $\textsc{ReadOut}$ operation that aggregates the representations of nodes in the subgraph $S_x$. That is,
\begin{align} \label{eq.readout}
    \vec{s}_{x} = \textsc{ReadOut}(\{\vec{h}_v:v\in V(S_x)\}).
\end{align}
The choice of the aggregation scheme for $\textsc{ReadOut}$ is flexible, including sum pooling and more advanced techniques \cite{ying2018hierarchical,xu2018powerful}. In our implementation, we simply use sum pooling.

In summary, the unification framework is enabled by the common task template of subgraph similarity learning, which lays the foundation of our pre-training and prompting strategies as we will introduce in the following parts.

\subsection{Pre-Training Phase}

As discussed earlier, our pre-training phase employs the link prediction task. Using link prediction/generation is a popular and natural way \cite{hamilton2017inductive,hu2020gpt,DBLP:conf/nips/HwangPKKHK20,lu2021learning},
as a vast number of links are readily available on large-scale graph data without extra annotation. In other words, the link prediction objective can be optimized on label-free graphs, such as those shown in Fig.~\ref{fig.framework}(a), in a self-supervised manner.

Based on the common template defined in Sect.~\ref{sec:model:unification}, the link prediction task is anchored on the similarity of the contextual subgraphs of two candidate nodes. Generally, the subgraphs of two positive ({\it i.e.}, linked) candidates shall be more similar than those of negative ({\it i.e.}, non-linked) candidates, as illustrated in Fig.~\ref{fig.framework}(b).
Subsequently, the pre-trained prior on subgraph similarity can be naturally transferred to node classification downstream, which shares a similar intuition: the subgraphs of nodes in the same class shall be more similar than those of nodes from different classes. On the other hand, the prior can also support graph classification downstream, as graph similarity is consistent with subgraph similarity not only in letter (as a graph is technically always a subgraph of itself), but also in spirit. The ``spirit'' here refers to the tendency that graphs sharing similar subgraphs are likely to be similar themselves, which means graph similarity can be translated into the similarity of the containing subgraphs \cite{shervashidze2011weisfeiler,zhang2018end,togninalli2019wasserstein}.

Formally, given a node $v$ on graph $G$, we randomly sample one positive node $a$ from $v$'s neighbors, and a negative node $b$ from the graph that does not link to $v$, forming a triplet $(v,a,b)$. Our objective is to increase the similarity between the contextual subgraphs $S_v$ and $S_a$, while decreasing that between $S_v$ and $S_b$. More generally, on a set of label-free graphs $\ensuremath{\mathcal{G}}$, we sample a number of triplets from each graph to construct an overall training set $\ensuremath{\mathcal{T}}_\text{pre}$. Then, we define the following pre-training loss.
\begin{align} \label{eq.pre-train-loss}
    \ensuremath{\mathcal{L}}_{\text{pre}}(\Theta)=-\sum_{(v,a,b)\in\ensuremath{\mathcal{T}}_\text{pre}}\ln\frac{\exp(\text{sim}(\vec{s}_v,\vec{s}_a)/\tau)}{\sum_{u\in\{a,b\}}\exp(\text{sim}(\vec{s}_v,\vec{s}_u)/\tau)},
\end{align}
where
$\tau$ is a temperature hyperparameter to control the shape of the output distribution. Note that the loss is parameterized by $\Theta$, which represents the GNN model weights.

The output of the pre-training phase is the optimal model parameters $\Theta_0=\arg\min_\Theta \ensuremath{\mathcal{L}}_{\text{pre}}(\Theta)$. $\Theta_0$ can be used to initialize the GNN weights for downstream tasks, thus enabling the transfer of prior knowledge downstream.

\subsection{Prompting for Downstream Tasks}

The unification of pre-training and downstream tasks enables more effective knowledge transfer as the tasks in the two phases are made more compatible by following a common template. However, it is still important to distinguish different downstream tasks, in order to capture task individuality and achieve task-specific optimum.

To cope with this challenge, we propose a novel task-specific learnable prompt on graphs, inspired by prompting in natural language processing \cite{brown2020language}. In language contexts, a prompt is initially a handcrafted instruction to guide the downstream task, which provides task-specific cues to extract relevant prior knowledge through a unified task template (typically, pre-training and downstream tasks are all mapped to masked language modeling). More recently, learnable prompts \cite{liu2021gpt,lester2021power} have been proposed as an alternative to handcrafted prompts, to alleviate the high engineering cost of the latter.

\stitle{Prompt design.} Nevertheless, our proposal is distinctive from language-based prompting for two reasons.
Firstly, we have a different task template from masked language modeling. Secondly, since our prompts are designed for graph structures, they are more abstract and cannot take the form of language-based instructions. Thus, they are virtually impossible to be handcrafted. Instead, they should be topology related to align with the core of graph learning. In particular, under the same task template of subgraph similarity learning,
the $\textsc{ReadOut}$ operation (used to generate the subgraph representation) can be ``prompted'' differently for different downstream tasks. Intuitively, different tasks can benefit
from different aggregation schemes for their $\textsc{ReadOut}$. For instance, node classification pays more attention to features that are topically more relevant to  the target node. In contrast, graph classification tends to focus on features that are correlated to the graph class. Moreover, the important features may also vary given different sets of instances or classes in a task.

Formally, let $\vec{p}_t$ denote a learnable \emph{prompt vector} for a downstream task $t$, as shown in Fig.~\ref{fig.framework}(c). The prompt-assisted $\textsc{ReadOut}$ operation on a subgraph $S_x$ for task $t$ is
\begin{align}\label{eq:prompt-fw}
\vec{s}_{t,x} = \textsc{ReadOut}(\{\vec{p}_t\odot\vec{h}_v:v\in V(S_x)\}),
\end{align}
where $\vec{s}_{t,x}$ is the task $t$-specific subgraph representation, and $\odot$ denotes the element-wise multiplication. That is, we perform a \emph{feature weighted} summation of the node representations from the subgraph, where the prompt vector $\vec{p}_t$ is a dimension-wise reweighting in order to extract the most relevant prior knowledge for the task $t$.

Note that other prompt designs are also possible. For example, we could consider a learnable \emph{prompt matrix} $\vec{P}_t$, which applies a linear transformation to the node representations:
\begin{align}\label{eq:prompt-lin}
\vec{s}_{t,x} = \textsc{ReadOut}(\{\vec{P}_t\vec{h}_v:v\in V(S_x)\}).
\end{align}
More complex prompts such as an attention layer is another alternative.
However, one of the main motivation of prompting instead of fine-tuning is to reduce reliance on labeled data. In few-shot settings, given very limited supervision, prompts with fewer parameters are preferred to mitigate the risk of overfitting. Hence, the feature weighting scheme in Eq.~\eqref{eq:prompt-fw} is adopted for our prompting as the prompt is a single vector of the same length as the node representation, which is typically a small number ({\it e.g.}, 128).

\stitle{Prompt tuning.}
To optimize the learnable prompt, also known as \emph{prompt tuning}, we formulate the loss based on the common template of subgraph similarity, using the prompt-assisted task-specific subgraph representations.

Formally, consider a task $t$ with a labeled training set $\ensuremath{\mathcal{T}}_{t} =\{(x_1,y_1),(x_2,y_2),\ldots\}$, where $x_i$ is an instance ({\it i.e.}, a node or a graph), and $y_i\in Y$ is the class label of $x_i$ among the set of classes $Y$. The loss for prompt tuning is defined as
\begin{align}
    \ensuremath{\mathcal{L}}_{\text{prompt}}(\vec{p}_t)=-\sum_{(x_i,y_i)\in \ensuremath{\mathcal{T}}_t}\ln\frac{\exp(\text{sim}(\vec{s}_{t,x_i},\tilde{\vec{s}}_{t,y_i})/\tau)}{\sum_{c\in Y}\exp(\text{sim}(\vec{s}_{t,x_i},\tilde{\vec{s}}_{t,c})/\tau)},
\end{align}
where the class prototypical subgraph for class $c$ is represented by $\tilde{\vec{s}}_{t,c}$, which is also generated by the prompt-assisted, task-specific $\textsc{ReadOut}$.

Note that, the prompt tuning loss is only parameterized by the learnable prompt vector $\vec{p}_t$, without the GNN weights. Instead, the pre-trained GNN weights   $\Theta_0$ are frozen for downstream tasks, as no fine-tuning is necessary. This significantly decreases the number of parameters to be updated downstream, thus not only improving the computational efficiency of task learning and inference, but also reducing the reliance on labeled data.

\section{Preliminaries}

In this section, we give the problem definition and introduce the background of GNNs.

\subsection{Problem Definition}\label{sec:prelim:problem}

\stitle{Graph.}
A graph can be defined as $G=(V,E)$, where $V$ is the set of nodes and $E$ is the set of edges. We also assume an input feature matrix of the nodes, $\vec{X}\in\mathbb{R}^{|V|\times d}$, is available. Let $\vec{x}_i\in\mathbb{R}^d$ denote the feature vector of node $v_i\in V$.
In addition, we denote a set of graphs as $\ensuremath{\mathcal{G}}=\{G_1, G_2, \ldots, G_N\}$.

\stitle{Problem.}
In this paper, we investigate the problem of graph pre-training and prompting. For the downstream tasks, we consider the popular node classification and graph classification tasks.
For node classification on a graph $G=(V,E)$, let $C$ be the set of node classes with $\ell_i\in C$ denoting the class label of node $v_i \in V$.
For graph classification on a set of graphs $\ensuremath{\mathcal{G}}$, let $\ensuremath{\mathcal{C}}$ be the set of graph labels with $L_i\in \ensuremath{\mathcal{C}}$ denoting the class label of graph $G_i\in \ensuremath{\mathcal{G}}$.

In particular, the downstream tasks are given limited supervision in a few-shot setting: for each class in the two tasks, only $k$ labeled samples ({\it i.e.}, nodes or graphs) are provided, known as $k$-shot classification.

\subsection{Graph Neural Networks}\label{sec:prelim:gnn}
The success of GNNs boils down to the message-passing mechanism \cite{wu2020comprehensive},
in which each node receives and aggregates messages ({\it i.e.}, features or embeddings) from its neighboring nodes to generate its own representation. This operation of neighborhood aggregation can be stacked in multiple layers to enable recursive message passing. Formally, in the $l$-th GNN layer, the embedding of node $v$, denoted by $\vec{h}^l_v$, is calculated based on the embeddings in the previous layer, as follows.
\begin{equation}
    \vec{h}^l_v = \textsc{Aggr}(\vec{h}^{l-1}_v, \{\vec{h}^{l-1}_u : u\in\ensuremath{\mathcal{N}}_v\}; \theta^l),
\end{equation}
where $\ensuremath{\mathcal{N}}_v$ is the set of neighboring nodes of $v$, $\theta^l$ is the learnable GNN parameters in layer $l$. $\textsc{Aggr}(\cdot)$ is the neighborhood aggregation function and can take various forms, ranging from the simple mean pooling \cite{kipf2016semi,hamilton2017inductive} to advanced neural networks such as neural attention \cite{velivckovic2017graph} or multi-layer perceptrons \cite{xu2018powerful}. Note that in the first layer, the input node embedding $\vec{h}^0_v$ can be initialized as the node features in $\vec{X}$.
The total learnable GNN parameters can be denoted as $\Theta=\{\theta^1, \theta^2, \ldots\}$. For brevity, we simply denote the output node representations of the last layer as $\vec{h}_v$.

\section{Related Work}

\stitle{Graph representation learning.}
The rise of graph representation learning, including earlier graph embedding \cite{perozzi2014deepwalk,tang2015line,grover2016node2vec} and recent GNNs \cite{kipf2016semi,hamilton2017inductive,velivckovic2017graph,xu2018powerful}, opens up great opportunities for various downstream tasks at node and graph levels.
Note that learning graph-level representations requires an additional $\textsc{ReadOut}$ operation
which summarizes the global information of a graph by aggregating node representations through a flat
\cite{duvenaud2015convolutional,gilmer2017neural,zhang2018end,xu2018powerful} or hierarchical \cite{gao2019graph,lee2019self,ma2019graph,ying2018hierarchical} pooling algorithm.
We refer the readers to two comprehensive surveys \cite{cai2018comprehensive,wu2020comprehensive} for more details.

\stitle{Graph pre-training.}
Inspired by the application of pre-training models in language \cite{dong2019unified,beltagy2019scibert} and vision \cite{lu2019vilbert,bao2022beit} domains, graph pre-training \cite{xia2022survey} emerges as a powerful paradigm that leverages self-supervision on label-free graphs to learn intrinsic graph properties.
While the pre-training learns a task-agnostic prior, a relatively light-weight fine-tuning step is further employed to update the pre-trained weights to fit a given downstream task.
Different pre-training approaches
design different self-supervised tasks based on various graph properties such as node features \cite{Hu2020Strategies,hu2020gpt}, links \cite{kipf2016variational,hamilton2017inductive,hu2020gpt,lu2021learning}, local or global patterns \cite{qiu2020gcc,Hu2020Strategies,lu2021learning}, local-global consistency \cite{velickovic2019deep,Sun2020InfoGraph,peng2020graph,hassani2020contrastive},
and their combinations \cite{you2020graph,you2021graph,suresh2021adversarial}.

However, the above approaches do not consider the gap between pre-training and downstream objectives, which limits their generalization ability to handle different tasks.
Some recent studies recognize the importance of narrowing this gap. L2P-GNN \cite{lu2021learning} capitalizes on meta-learning \cite{finn2017model} to simulate the fine-tuning step during pre-training. However, since the downstream tasks can still differ from the simulation task, the problem is not fundamentally addressed.
In other fields, as an alternative to fine-tuning, researchers turn to prompting \cite{brown2020language}, in which a task-specific prompt is used to cue the downstream tasks. Prompts can be either handcrafted \cite{brown2020language} or learnable  \cite{liu2021gpt,lester2021power}. On graph data, the study of prompting is still limited. One recent work called GPPT \cite{sun2022gppt} capitalizes on a sophisticated design of learnable prompts on graphs, but it only works with node classification, lacking a unification effort to accommodate other downstream tasks like graph classification.
Besides, there is a model also named as GraphPrompt \cite{zhang2021graphprompt}, but it considers an NLP task (biomedical entity normalization) on text data, where graph is only auxiliary. It employs the standard text prompt unified by masked language modeling, assisted by a relational graph to generate text templates, which is distinct from our work.

\stitle{Comparison to other settings.}
Our few-shot setting is different from other paradigms that also deal with label scarcity, including semi-supervised learning \cite{kipf2016semi} and meta-learning \cite{finn2017model}. In particular, semi-supervised learning cannot cope with novel classes not seen in training, while meta-learning requires a large volume of labeled data in their base classes for a meta-training phase, before they can handle few-shot tasks in testing.

redpajama_set_name: RedPajamaArXiv


text:
$17 Billion Aid Deal for Ukraine Gets Green Light
International Monetary Fund Managing Director Christine Lagarde at IMF Headquarters in Washington, April 30, 2014.
Cliff Owen—AP
By Charlie Campbell
A $17 billion aid program has been approved for cash-strapped Ukraine, as the Kiev government struggles to maintain control of restive eastern regions near the Russian border.
The two-year deal will see the International Monetary Fund (IMF) immediately plow in $3.2 billion needed to prevent the embattled government from defaulting on its debts, as the East European nation teeters on the brink of bankruptcy.
The loan will give Ukraine vital breathing space from international creditors and allow the nation to tackle a $2.2 billion debt owed to Moscow for natural gas. Russia has been threatening to turn off supplies, further exacerbating tensions.
"Today's final approval for the $17 billion IMF program marks a crucial milestone for Ukraine," U.S. Treasury Secretary Jack Lew said in a statement.
IMF chief Christine Lagarde said the organization would keep a close watch to ensure the Kiev government abides by the terms of the deal.
The loan also paves the way for Ukraine to receive an additional $15 billion in funding pledged by the World Bank, Japan, the E.U., Canada and the U.S.
On Monday, the Obama Administration leveled more sanctions against individuals and companies close to Russian President Vladimir Putin, as part of a "calibrated effort" to prevent the Kremlin fostering unrest in its southwestern neighbor.
And Lew warned on Wednesday that the U.S. would "impose increasing costs on Russia if it persists in its efforts to destabilize Ukraine."
Write to Charlie Campbell at charlie.campbell@time.com.

redpajama_set_name: RedPajamaCommonCrawl


text:
Season Premiere: What should we expect from post-Wrestlemania call-ups and returns?
styllyn24	Uncategorized	 April 15, 2018 4 Minutes
Welcome back! After attempting to pound out decent Wrestlemania week content every day, it was time for a break. Now here we are again, it's a new "season" of WWE with new and returning characters. Let's take a look at who is new, who is back, and what each one brings to the table, starting with the Raw side.
Ember Moon is awesome. I've loved her since she debuted in NXT, which I witnessed live at Takeover Brooklyn II (I'm sure she wrestled at house shows before this, let's not get technical here). Her NXT feud with Asuka is one of the best feuds in recent memory, and the matches were incredible. It would seem that Asuka is moving to Smackdown and Ember will likely stay on Raw, but hopefully at some point down the road we get to see him go at it again. I feel like Smackdown would be a better fit for her, I would love to see the matches she could put on with Asuka, Charlotte, and Becky Lynch. I'm not quite sure where she fits into the Raw women's roster. Maybe those will both be shaken up this week, and Becky and Charlotte move to Raw. As the rosters stand, I could see a good feud with Sasha or Alexa. I would think she starts off wrestling the non-main event women's talent like Mickie James. Ember has the in-ring ability and the charisma to be a pillar of the women's division for a long time.
I think No Way Jose is going to be used just like Adam Rose. A fun entrance, matches that don't showcase his upper-quality in-ring ability, a badly-executed heel turn, and ultimately a release (though hopefully not due to DV arrest). He's a very talented wrestler and oozes charisma, but unfortunately I don't see much of a future for the gimmick as a face or a heel. Enjoy him while he lasts and hope that I'm wrong.
Hopefully Jeff can keep his personal life in order after his latest DUI to stick around for a while. I'd like to see him join up with Matt and Bray, I'm really starting to enjoy this Woken/Broken Universe and am very intrigued to see how Bray ties into it, and adding Jeff into that mix would be cool. I'm not sure exactly what he would do as a singles competitor at this time, at least not on Raw. If he's not going to be part of the team, a move to Smackdown may be the best thing for him. I think he could have really good matches with guys like AJ and Shinsuke.
Wasn't impressed with Bobby Lashley before, and I doubt I'm going to be impressed with him now. This really doesn't do anything for me. I think of Bautista's last WWE run when I think about how he's going to be used and how it's going to end. In other words, not good. Again, I hope I'm wrong, but I just don't see a ton of potential positives here.
When thinking about different directions for AOP, I think of two possible outcomes. First, and the one I hope for, is the Bludgeon Brothers. Dominating, brutal, squashing opponents, tag team champions. People just want to see them destroy and conquer. Second is the direction of the team that dominated NXT more than any other tag team then hit a brick wall on the main roster: The Ascension. Their segments with Breezango were entertaining, but that team was ruined. They were brutal destroyers in NXT, similar to the Authors of Pain, but now have been reduced to comedy act jobbers. Obviously these are two extremes, and AOP could end up somewhere in the middle, but I'd like to see them used more like the Bludgeon Brothers. Don't be surprised to see some squash matches against jobbers for a few weeks, but I expect to see them in the tag team title mix sooner rather than later.
We have seen what Samoa Joe brings to the table. I expect his match with Roman at Backlash to be for the Universal Championship after Roman takes it from Brock in Saudi Arabia. If he stays healthy, which is a big IF, I think he will be in the main event picture if not Universal Champion for most of this year. He and Roman seem to work well together and their promos seem personal. People get behind him even though he's a heel because he's just brutal in the ring and has a ton of charisma. I wonder how much he has left with him seemingly injured fairly often and also about to turn 40. This may be his last run.
Taking a look at the Smackdown side, here's the perfect segue:
Paige retires on Raw, and one night later is named the new GM of Smackdown. It is a cool moment and obviously mirrors Daniel Bryan's path. I honestly have no idea how this is going to work out. She's good on the mic, and I don't know if she'll work better as a face or heel. I have a feeling a turn is coming, we've spent the last year with face authority figures on both shows for the most part, and usually that changes every few months or so. Let's see where this goes.
No. Next.
Seriously, I hate these girls. They're obnoxious. Maybe it'll be fun to see them get their faces smashed in, but I doubt that's what will happen. They'll win and win and win and I'll be forced to eat cyanide from listening to them brag about it. Make it stop.
I still expect to see Sanity debut and Big Cass return very soon. Until then, what do you want to see from these returnees/new arrivals?
Previous Post Where Does This Past WrestleMania Fit?
Next Post Raw Review: The Superstar Shakeup Show
One thought on "Season Premiere: What should we expect from post-Wrestlemania call-ups and returns?"
Pingback: Raw Review: The Superstar Shakeup Show – It's Damn True: All about WWE


text:
Coombe Green, Worcestershire
Last weekend I pulled up and photographed a building I'd been meaning to post for a while, the Mission Room on the edge of the common at Coombe Green in Worcestershire. It seemed to me as I looked at it that I'd better share it with you soon. Many of these "tin churches" are disappearing, and when I see one that's not in pristine condition, I wonder whether it, too, will be pulled down to make way for something else.
According to a notice in the window, this corrugated-iron church was licensed for worship in 1904. As I looked at it, I wondered whether the original builders thought that it would last more than a hundred years. It's actually quite a complex building for a tin church – many consist simply of four sides and a roof, perhaps with the addition of a small porch. This one has what looks like a separate chancel, just visible to the left, plus a transept-like extension also on the left, the little store room on the right, and the long porch, its roof held up with a row of rugged tree trunks, that runs along the front. There's a even a bell turret with a tiny spire-like roof.
These churches were often bought as prefabricated buildings from big suppliers, but this one, with its tree-trunk supports, might be custom-made. That porch gives the building an individual, frontier-town character, something that's in keeping with this part of Worcestershire – not far away is a house made partly from an old railway carriage and a common with a number of scattered houses, probably put up by their original owners to take advantage of the old law that said that if you could build a house in a single night on a piece of unoccupied land, you gained the right to stay there. Rugged individualism indeed.
Posted by Philip Wilkinson at 8:37 AM 12 comments:
Labels: church, common, Coombe Green, corrugated iron, mission room, tin tabernacles, Worcestershire
Newell Street, London
In the dappled green shade of the churchyard of St Anne's, Limehouse (see previous post) is a 9-foot-high stone pyramid. It seems to mark no grave, has no obvious purpose, and stands there, looking mysterious. In 19th-century engravings it is shown mounted on a square plinth but now stands on the ground. On one face is the inscription, 'The wisdom of Solomon'. The pyramid is part of a group of arcane symbols that seem to cluster around Hawksmoor's churches – pyramids, obelisks, and other borrowings from ancient civilizations – that set them apart and contribute to the alien character of these massive buildings. The best guess about the surprising presence of this particular pyramid is that it was intended to top one of the corners of the church. No one knows for sure.
Such things arouse the interest of psychogeographers, those writers who have examined the lives of places and their effects on people's psyches. Mappers of hidden cultural contours; those who exhume forgotten histories or find alignments between the past and the present; soakers in atmosphere; reclaimers of evocative desolation and pleasing decay.
Iain Sinclair, for example. A whole section of Sinclair's book Lud Heat (part poetry, part prose, 'a writing' as that other great London visionary, David Jones, would have called it) concerns Hawksmoor's churches. It includes a map with lines that join their sites, creating triangles and pentagrams and alignments with other highly significant sites and buildings such as William Blake's house in Lambeth and Cleopatra's Needle. Sinclair mines all kinds of allusions, myths, and significances from these alignments, connections that enable him to incorporate into his text references to such writers as Blake, Pepys, and Sir Thomas Browne, parallels with myths from ancient Egypt and ancient Britain, and sinister histories ranging from the plague to the murders committed by Jack the Ripper.
Who knows how significant all this is? It's possible to make all sorts of sites connect on a map, especially if the scale of the map is small and the lines joining the points are thick, as critics of ley-line hunters have pointed out. But I'm not looking for literal truth or Ordnance Survey precision in the alignments between pyramids, obelisks, and plague pits. What is more important is that Sinclair has built absorbing and sometimes brilliant writing out of this hoard of images and myths.
And Sinclair nails Hawksmoor's style. He says of the churches: 'Certain features are in common: extravagant design, massive, almost slave-built, strength – not democratic. A strength that is not connected to notions of "craftsmanship" or "elegance". They are not easy on the eye, and do not enforce images of grace. Metaphors inflate at their own risk. The mind is not led upwards to any starry nest.' Sometimes Sinclair could almost be describing his own writing here, prose that has a strangeness and a solidity that is true to Hawksmoor's buildings, large, ponderous, and weird as they are. Words that are arranged in a modern, collage-like fashion, and yet draw on ideas that have been around for as long as the pyramids of Egypt.
Posted by Philip Wilkinson at 3:39 PM 8 comments:
Labels: alignments, church, Georgian, Hawksmoor, Iain Sinclair, London, Lud Heat, psychogeographers, pyramid, Solomon, St Anne's Limehouse
Landmark and sea mark
St Anne's Limehouse, built between 1714 and 1727, is among the remarkable London churches designed by Nicholas Hawksmoor. It is one of 12 churches built at this period and funded by a tax on coal coming up the Thames and these buildings are now known as Queen Anne churches, in honour of the reigning monarch of the time.
St Anne's is probably less well known than Hawksmoor's central London churches, such as St Mary Woolnoth or Christ Church Spitalfields (also a Queen Anne church), buildings that loom out of the city in a way that it's impossible to ignore. St Anne's, by contrast, is slightly less monumental, and is quite easy to miss in its leafy churchyard on Commercial Road. But the tower is a stunner, and Hawksmoor obviously designed it to be seen not from the main road but at the end of this little side street.
The way the very angular tower relates to the small half-domed vestibule and then steps back as it rises, is striking. The semi-circular arch beneath the clock, and the tiny semi-circular opening lower down, harmonize with the curving half-dome. But the rest of the tower is much more straight-edged, with its bold corner pilasters and unusual octagonal lantern at the top above the clock. This lantern has a set of pinnacles, just visible in my photograph, that are topped with tiny pyramids and look like miniature Hawksmoor towers.
This artful tower had a purpose beyond the usual one of housing bells. The church is not far from the river, and at the very top, higher even than the pinnacles, is a flagpole supporting a golden ball. This acted as a sea mark for shipping, and seamen were also aided by the clock (said to be the highest church clock in London), which originally struck every quarter. Even such off-the-wall structures as Hawksmoor towers had their practical uses.
More soon on St Anne's churchyard, with its rather spooky pyramid. Watch this space…
Posted by Philip Wilkinson at 2:59 PM 14 comments:
Labels: church, eighteenth-century architecture, Hawksmoor, Limehouse, London, Newell Street, Queen Anne, sea mark, St Anne's Limehouse, Thames, tower
Winchester, Hampshire
A coat of arms
On several occasions on this blog I've noticed premises belonging to W H Smith, and looked at the various telling ways at which this company, one of the first multiple retailers, has decorated its shops over the years. I've looked at tiles, hanging signs, and even drainpipes. This time, it's heraldry, and my post is as much a query as an observation.
A while back I passed the branch of Smith's in the centre of Winchester and admired this coat of arms adorning the corner of the shop. I'm often struck by the ways in which buildings turn corners, and this bit of corner colour naturally caught my eye. But no one could tell me whose coat of arms this is, or was, and why it's here. I'd love to know the meaning of the three roses, why the supporting lions are standing on the sterns of sailing ships, and why the female figure, presumably Justice with her sword and scales, is included at the top. There was no one around to ask about this on the quiet evening when I passed last year, and I can find no similar arms online. If anyone has the answer, I'd be fascinated to know.
The answer has come. These appear to be the arms of Southampton. Many thanks to the anonymous commenter who provided the information and link.
Labels: coat of arms, Hampshire, heraldry, shop, W H Smith, Winchester
Solihull, Warwickshire
Greece in the Midlands
This is the most unexpected sight to see on a suburban street in Solihull. Its architect, the young John Soane, soon to design the most elaborate country houses, such as Wimpole Hall in Cambridgeshire and Tyringham Hall in Buckinghamshire, described it as a barn à la Paestum. While Soane's houses are elegant Georgian or Regency buildings with complex interiors often displaying a sense of space that's both intricate and dazzlingly sophisticated, this barn is designed in the simplest, plainest red-brick classicism.
The building dates to the early phase of the architect's career. He had studied at the Royal Academy and in 1777 won the Academy's most glittering prize: its Travelling Scholarship. This enabled him to go to Italy to study the buildings of Rome. While there he also traveled south to visit Caserta, Baia, and Pompeii. One of the highlights of this trip was Paestum, the Greco-Roman city some 50 miles southeast of Naples, where the ruins of three ancient Doric temples, built by Greek colonists, survive. These temples, dating from the 5th and 6th centuries BC, are among some of the best preserved of all ancient Greek buildings. Their antiquity and their proportions inspired Soane.
While in Italy Soane met a number of British aristocrats and gentry whom he saw as prospective clients. Among them were several young men who hired Soane as a draughtsman to travel with them from Italy to Sicily and Malta. One member of this group was Henry Greswold Lewis, a landowner from Solihull, and it was for Lewis in 1798 that Soane designed this unlikely structure as a kind of homage to the ruins he had seen at Peastum.
The building is severe, and rather shocking. It's not exactly like a Greek temple, which would have evenly spaced columns, a deeper entablature above the columns, and no round arch in the centre. But its carefully laid soft red bricks and imperfect white entablature do the job of suggesting ancient Greece and translating some of its architectural hallmarks into the very English medium of brick. For all its severity, it must have made Lewis think of his youthful travels, and smile.
Labels: barn, brick, Doric, Georgian, Paestum, Regency, Soane, Solihull, Warwickshire
Fulham High Street, London
Still there – just
It must have been in the 1970s, when Philip Larkin's Oxford Book of Twentieth Century English Verse came out, that I first read and was moved by a poem by P J Kavanagh called 'The Temperance Billiards Rooms'. In it, the poet remembers how he used to walk past the Temperance Billiards Rooms with his wife, who died tragically young. Aged just 33, the poet salutes the Billiards Rooms alone. He makes the building stand for continuity, in this moving poem about carrying on after a disaster: 'it just goes on, as I do too I notice'. But it's also fragile ('something so uneconomical's sure to come down') and so is the grieving poet. It's a touching poem, and Kavanagh's description of the place, 'in red and green and brown, with porridge-coloured stucco in between and half a child's top for a dome…it's like a Protestant mosque!' has a melancholy humour.
I wonder if this is Kavanagh's building. It's now a pub (The Temperance), is repainted a rather sorry dark grey and is offering special deals on cocktails. There's still a dome, still bits of stucco decoration, still stained glass in red and green, still the hall to the right which must have contained the billiard tables. If it's not the building in the poem, it's one very like it. Designed in 1910 by Norman Evans, it was one of several such buildings, built for Temperance Billiard Halls Ltd in northern England and the suburbs of London. Their art nouveau glass and decoration, and the chance of a game of billiards, were meant to attract punters away from the temptations of the demon drink, part of a movement that began in the middle decades of the 19th century and saw a late resurgence in 1900–1910. As late as 1908 there was a bill in Parliament to reduce the number of licenses to sell alcohol and ban the employment of women in pubs, a bill that was vigorously supported by temperance campaigners, and equally loudly decried by others, especially those, such as the Barmaids' Political Defence League, who stood up for the barmaids who were to lose their jobs. The bill was defeated in the end, and the temperance movement declined.
This billiards hall, at any rate, is still there, although the dark paint spoils what looks it had. It's also fiendishly difficult to photograph, hemmed in by road signs, wires, aerials, railings, and continuous traffic along the Fulham High Street. The best I could do was to include one of the most interesting vehicles that went past as I stood outside, a Mercedes Benz 280SL that takes us just about back to the 1960s, when Kavanagh wrote his poem. Like him, I'm glad the building is still there, although I cannot, like the poet, say that there are, 'for all I know men playing billiards temperately in there'.
For more about architecture and temperance, see Andrew Davison's essay '"Worthy of the Cause": The Buildings of the Temperance Movement' in Geoff Brandwood (ed), Living, Leisure and Law (Spire Books, 2010).
For P J Kavanagh's account of his loss, see P J Kavanagh, The Perfect Stranger (1966)
Labels: art nouveau, billiards, Fulham, High Street, London, P J Kavanagh, pub, stucco, temperance
Birtsmorton, Worcestershire
History and harmony
If you said the words "romantic moated manor house" a building like Birtsmorton Court might well come to mind. A structure that has evolved over seven or eight centuries, this beautiful house is made up of a mixture of timber-framed, stone, and brick wings, all different but wonderfully harmonious, set in quiet countryside, and partly shaded by trees. Birtsmorton may have been begun in 13th century, but in the 15th it was bought by one John Nanfan, who rebuilt most of it. In subsequent centuries, the house was home to various landed families (including relatives of Richard Hakluyt, the writer on exploration, and the family of William Huskisson, the statesman who was the first person to be killed by a railway train) and several of these later occupants made substantial alterations to the building, producing the delightful hotchpotch that remains today.
My photograph shows the view from the south, where there are buildings of various periods in different materials. On the right, the house is stone below, timber-framed with brick infill between the timbers above; this timber-work is a replacement of 1929–30 of earlier work that had been destroyed by fire. On the left is a contrasting brick wing built in the 18th century, with sash windows. Between these two parts is a mixture of various dates, with a pair of timber-framed bays, some stone walls, and tall brick chimneys.
It would take a long time to unpick the complex architectural history that produced this rich and diverse collection of walls, gables, roofs, and chimneys. Even for someone with expertise in the archaeology of buildings it would be a challenge, and one would need to have the run of the place for some considerable time. As Birtsmorton is still a private house, archaeologists are not likely to be unpicking it any time soon. Even if its detailed history is hard to decipher, though, its visual harmony is intact, and a thing to marvel at.
The house is available for events such as weddings, and there's more information about it here.
Labels: Birtsmorton, Hakluyt, house, Huskisson, Manor, moat, timber-framed, Worcestershire
The Buttercross, before...
Last time I was in Ludlow, which was last year, I noticed the way the stone of the Buttercross, a building I'd long admired, was glowing in the sun, and took the photograph above. This building, part market, part Town Hall, was built in 1743–4 to designs by William Baker (1705–71), an architect who was busy in Shropshire, Staffordshire, and neighbouring counties in the middle of the 18th century. Pevsner, describing the architecture of the building, says while it is "not polished, [it] has an attractive robustness". The classical portico with its four Tuscan columns is offset by less formal details, such as the semicircular window and the clock turret, with its lovely cupola. The whole thing is a worthy centrepiece to this part of the town centre, and looks good from Broad Street, where I took the picture.
Since last year, the builders have been in, doing major works to the roof. Then, in November, disaster struck in the form of falling chunks of plaster from one of the ceilings. Further internal problems have since been discovered and Ludlow's council have a much bigger and more complex restoration project on their hands. The plaster seems to have come down without much provocation. According to the Shropshire Star, Ludlow's Mayor, Martin Taylor-Smith, said, "We think the initial fall was when the clock-winder went up, just from the vibration."
Now the Town Council, which used the upper part of the building, has found alternative accommodation, work has begun on finding a new use for the Butter Cross. An application in underway to turn the building into an education and interpretation centre, where, for example, traditional building skills might be shown. Here's hoping the extra funding can be found to make the repairs and preserve this landmark.
Labels: buttercross, Georgian, Ludlow, market, Restoration, Shropshire, town hall, traditional building skills
Much Marcle, Herefordshire
On the curve
I remember a few years ago a conversation with a friend about garages and what they look like, how most of them are either very boring or very unpleasant to look at, but now and then, one stands out from the crowd. Before long, one of us said, 'Do you know the garage on the road between Ledbury and Ross on Wye, at Much Marcle?' and the other one instantly said, 'Yes! Isn't it terrific!' We'd both been admiring the building for years, and I can't remember which of us mentioned it first. There are several things I like about it. The way it stands at a slight angle to the junction. The gentle curve of the roof, a curve followed by the attractive lettering on the front. The mixture of corrugated iron and wood. The building began life as a World War I aircraft hangar. It was bought by the Weston's Cider Company, who are based nearby, in 1926, and they used it to maintain their vehicles as well as offering a general garage service. In the 1990s, Weston's sold it, and it continues as a garage serving the general public.
Not everyone admires this kind of thing, of course. It doesn't happen often that I find myself at odds with the Shell Guides, old books that I admire because they still have a lot to tell us about architecture and the sense of place. In the 1955 Herefordshire guide, author David Verey found much to like in Much Marcle, but his admiration was 'in spite of its approach from the Ledbury road being marked by an ugly new garage'. Verey couldn't wait to get on to the village's old church and houses, the place's polite architecture, as they say. I, on the other hand, wanted to linger here on the main road, taking in this small landmark as the motorcyclists whizzed by enjoying the challenging mix of bends and straights on the way to Ross and perhaps themselves registering, through an eye corner, a curving metal roof and a painted garage sign.
Footnote: Garagistes may like another post that I did a while back, about two garages in Upton on Severn.
Labels: cider, corrugated iron, garage, Herefordshire, Ledbury, Much Marcle, Ross-on-Wye, Shell Guides, Verey


text:
package org.openforis.collect.io.metadata;

public enum ReferenceDataExportOutputFormat {

	CSV("csv"), EXCEL("xlsx");

	private String fileExtesion;

	private ReferenceDataExportOutputFormat(String fileExtesion) {
		this.fileExtesion = fileExtesion;
	}

	public String getFileExtesion() {
		return fileExtesion;
	}
}

redpajama_set_name: RedPajamaGithub


text:
Hamilton and Mercedes nominated for Laureus World Sports Awards
15 January 2020 Sam Hall F1 0
Lewis Hamilton and Mercedes have received nominations for the Laureus World Sports Awards after another successful campaign in 2019.
Somewhat surprisingly given his level of success in Formula One, especially in more recent years, Hamilton has never been the recipient of a Laureus World Sports Award.
Article continues under video
Mercedes, nominated for the team of the year award, have twice taken home the trophy, winning in 2018 for their 2017 efforts and in 2010 for successes while known as Brawn GP.
Former two-time Formula One world champion and Laureus Academy Member Mika Hakkinen said, "Lewis continues to dominate Formula One with the excellent Mercedes car he is driving. They both deserve their nominations. Whatever changes the sport makes, Lewis and Mercedes always seem to adapt.
"Congratulations to both of them. The big question everyone wants to know now is whether Lewis can pull level with Michael Schumacher on seven world titles next year. At the moment you would have to say it seems likely. It will make it an interesting year."
Also nominated for the best comeback of the year is German Sophia Floersch. After crashing heavily in the 2018 Macau Grand Prix, Floersch returned to compete one year later.
Azerbaijan GP worth $500m to local economy
Russell hopes to truly be in 2020 fight


Chinese Internet payment platform Alipay will launch a travel value-added tax refund service in ten countries in Europe to allow consumers to complete tax refunds by scanning codes with its app instead of waiting in line at the refund counters.
With this service, Chinese tourists will be able to realize tax refunds via code scanning and the refund money will be transferred to their Alipay accounts in Chinese Yuan.
Due to complicated procedures and slow payments into their accounts, some Chinese tourists in the past gave up their tax refund opportunities. Chinese media reports state that Chinese tourists lose nearly CNY1 billion in tax refunds in Europe annually.
According to Alipay, its new service may alleviate these problems. The service is jointly launched by Alipay and Global Blue, the world's largest tax refund organization; and it is expected to expand into 23 countries in Europe.
At present, this service is available at Global Blue's tax refund counters in Milan Malpensa Airport, Munich Airport, Berlin Tegel Airport, Rome Fiumicino Airport, and Helsinki-Vantaa Airport. It is expected to be available in Britain and France this summer.


I have sold a property at 172 Highland PL in Penticton.
GOING TO COURT APRIL 23: Call us for details. CONTINGENT upon court approval. Great opportunity here! Very rentable half duplex that only needs some minor TLC. 3 beds, 2 bathrooms, a decent rec room, separate laundry room, large deck with western exposure, decent back yard for kids and pets, and extra parking for toys, are just some of the features of this home. Only minutes to schools and shopping located on a quiet cul-de-sac. All offers must include a "Schedule A" obtained from listing representatives. Call for further details and court processes.


text:
Williamstown maintains same levy rates
Madeline Scarborough
mscarborough@newsandsentinel.com
Williamstown City Council members Jim Stage and Marty Seufer discuss the 2015 International Property Maintenance Code for health and safety of the town. (Photo by Madeline Scarborough)
WILLIAMSTOWN — The city's levy rates for 2019-2020 will stay the same as last year, council member Marty Seufer announced at Tuesday evening's Williamstown City Council meeting.
Council also approved the start of construction on a building for the community to use for baseball and softball practice during rainy and cold weather.
"The building will be out near the ballfields," said council member Jim Stage.
Stage said the building won't cost the city any money to build.
"Dr. Steven Eddy, a local chiropractor, and other businesses or locals have donated everything from materials and money to labor," he said.
The 40-foot by 80-foot building will have bathrooms and an open area for batting practice, pitching practice and exercising, Stage said. It is unknown when the building might be completed.
Also discussed at the council meeting were the 2015 International Property Maintenance Code for health and safety of the town and the no parking ordinance.
Council passed the first reading to adopt chapter one of the maintenance code.
Building official David VanHorn originally brought the issue to council's attention in March, saying he would like council to adopt the code.
"I have the same maybe eight people who blow grass into the streets and maybe eight who don't pick up their trash," VanHorn said.
VanHorn said adopting the code would give the city a process to follow to enforce the rules.
The board of appeals for the code will be made up of three people from the board of zoning and three volunteers, Stage said. According to Seufer, the fee will be "no less than $50 and no more than $500 per occurrence, after a citation has been issued."
Williamstown City Attorney Blaine Myers will be writing a civil infraction for the code.
The no parking ordinance is being drafted by Myers and the city will vote on it at the next meeting.
"There are many streets that are too narrow for cars to park safely on both sides with the amount of traffic that travels on it," Stage said.
Arbor Day will be celebrated April 26 at 1:30 p.m. at Tomlinson Park.
Copyright © News and Sentinel | https://www.newsandsentinel.com | 519 Juliana St., Parkersburg, WV 26101 | 304-485-1891 | Ogden Newspapers | The Nutting Company