text:
Map showing the underwater topography (bathymetry) of the ocean floor. Like land terrain, the ocean floor has ridges, valleys, plains and volcanoes. Image Credit: Public Domain, https://commons.wikimedia.org/w/index.php?curid=617528
Posted on April 8, 2020 April 8, 2020 by Evan Gough
Seriously, Life Really Does Get Around. It was Found in Rocks Deep Beneath the Seafloor
After a lot of hard work spanning many years, a team of scientists have discovered something surprising. They've found abundant bacterial life in tiny cracks in undersea volcanic rock in the Earth's crust. The bacteria are thriving in clay deposits inside these tiny cracks.
This discovery is generating new excitement around the hope of finding life on Mars.
The team of researchers presented their results in a new paper titled "Deep microbial proliferation at the basalt interface in 33.5–104 million-year-old oceanic crust." Lead author of the paper is Yohey Suzuki, associate professor at the University of Tokyo. The paper is published in the journal Nature Communications Biology.
"I was very lucky, because I almost gave up."
Yohey Suzuki, Lead Author.
The scientists found the single-celled creatures in basaltic lava deep beneath the sea floor. It took them over a decade to come up with a way to find these creatures in drill samples. They estimate that the rock cracks contain a density of 10 billion bacterial cells per cubic centimeter, which is about as dense as the human gut. For contrast, there are only about 100 cells per cubic centimeter in the mud on the seafloor.
The bacteria appear to be thriving in the clay that fills the tiny cracks. According to lead author Suzuki, anywhere you find clay, you find life.
"These cracks are a very friendly place for life. Clay minerals are like a magic material on Earth; if you can find clay minerals, you can almost always find microbes living in them," explained Suzuki in a press release.
The team spent a decade looking for life in the volcanic rock. They found it in rock core samples collected in 2010 as part of the Integrated Ocean Drilling Program (IODP). During their expedition, the ship they were on collected samples at three separate sites in the South Pacific Gyre. They used a metal tube that was 5.7 km (3.5 miles) long just to reach the ocean floor. Then they drilled another 125 meters into the ground—the first 75 meters were mud before they reached the bedrock itself.
"I thought it was a dream, seeing such rich microbial life in rocks."
One of the keys to understanding the significance of this discovery is the location of the samples. They weren't anywhere near any thermal vents or sub-seafloor water channels. That means the bacteria couldn't have been forced into the cracks by currents. It had to be living there. There were also three ages of rock samples in the study: 13.5 million, 33.5 million and 104 million years old.
Suzuki and the team had examined the rock samples many times, using established methods of looking for life, but found nothing. Suzuki and the other researchers eventually developed a new method for looking for it. They used a process similar to how pathologists cut laser-thin slices of tissue to examine it. That went against the grain of the standard way of looking for life in rock samples: grinding it up into a powder and counting cells.
One of the thin slices of rock prepared for examination through a microscope. Image Credit: Caitlin Devor, University of Tokyo, CC BY 4.0
After the thin slices were stabilized with epoxy, they were treated with a dye that stains DNA. Then they were examined with a microscope. Under the microscope, the bacteria appeared as green glowing spheres packed inside tunnels and cracks that glowed orange. By analyzing the DNA, the team identified the different types of bacteria living in the clay-filled cracks. The samples contained similar but not identical bacterial species.
When they finally found the evidence of life in these cracks, Suzuki was surprised.
Aerobic bacteria live densely packed into tunnels of clay minerals found in this sample of solid rock, collected from 122 meters beneath the seafloor. Image B is 1,000 times greater magnification than Image A. The left side photo in each image was taken using normal light and the right side photo was taken using fluorescent light. The solid basalt rock is gray, the clay minerals are orange, and the bacterial cells are green spheres. Image Credit: Suzuki et al. 2020, DOI: 10.1038/s42003-020-0860-1, CC BY 4.0
"Honestly, it was a very unexpected discovery. I was very lucky, because I almost gave up," said Suzuki. "I thought it was a dream, seeing such rich microbial life in rocks," he said.
"These cracks are a very friendly place for life. Clay minerals are like a magic material on Earth; if you can find clay minerals, you can almost always find microbes living in them," explained Suzuki.
While exciting in its own right, this discovery may be significant when it comes to the search for life on Mars. There are abundant clay minerals on Mars in areas that were once wet. The minerals in some of these clays could only have formed in the presence of water. The next rover to visit Mars, NASA's Mars 2020 Perseverance Rover, will sample some of that material. Hopefully, some of those samples will eventually be returned to Earth.
The proposed Mars Sample Return missions would send a spacecraft to Mars to gather samples collected by the Mars Perseverance (2020) Rover and then return them to Earth. Image Credit: By NASA/JPL-Caltech – https://photojournal.jpl.nasa.gov/jpeg/PIA23496.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=86914810
"I am now almost over-expecting that I can find life on Mars. If not, it must be that life relies on some other process that Mars does not have, like plate tectonics," said Suzuki. There's no definitive evidence that Mars has plate tectonics, though some satellites have detected magnetic anomalies that could be related to plate tectonics.
But there are many similarities between the types of rocks sampled in this study, and the types of rocks on Mars. They share many of the same conditions that formed them independently on each planet.
"Minerals are like a fingerprint for what conditions were present when the clay formed. Neutral to slightly alkaline levels, low temperature, moderate salinity, iron-rich environment, basalt rock — all of these conditions are shared between the deep ocean and the surface of Mars," said Suzuki.
The discovery of extremophiles here on Earth has injected some excitement into the search for life on other worlds. If life can survive in extreme environments on Earth, maybe it can survive in some of the extreme environments found throughout the Solar System.
The otherworldly landscape of Dallol, Ethiopia. This is the only place on Earth where scientists have found water, but no life. Extremophiles have found a way to thrive in other extreme environments on Earth, which has generated optimism around finding life in the Solar System. Image Credit: By Kotopoulou Electra – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=74975209
Results like the ones in this study are also fueling the excitement, at least for Suzuki. And that's leading to some collaboration between Suzuki and his team, and NASA. The researchers are going to help develop a plan for how to examine rock samples collected by rovers on Mars.
As the authors write in the conclusion of their paper, "The results of this study also have implications for the possibility of life on Mars and other planetary bodies. Basaltic crust is ubiquitous on other planets, such as Mars, as well on Earth." They also say "Given the subsurface presence of methane and liquid water on Mars, the communities fueled by organic matter and methane in subseafloor basalt on Earth provide a clear model for extant life and/or biosignatures from past life in the subsurface of Mars and other planets."
"This discovery of life where no one expected it in solid rock below the seafloor may be changing the game for the search for life in space," said Suzuki.
Press Release: Discovery of life in solid rock deep beneath sea may inspire new search for life on Mars
Research Paper: Deep microbial proliferation at the basalt interface in 33.5–104 million-year-old oceanic crust
Universe Today: Nutrient-Poor and Energy-Starved. How Life Might Survive at the Extremes in the Solar System
CategoriesEarth, Mars, Science Tagsbacterial life, Earth, extremophiles, life, Mars, ocean floor
One Reply to "Seriously, Life Really Does Get Around. It was Found in Rocks Deep Beneath the Seafloor"
Torbjörn Larsson says:
I was surprised that they found mostly aerobic prokaryotes, and heterotrophs at that. So while it isn't a nice fit to e.g. Mars, it is promising.
Previous PostPrevious Study Finds Bizarre Exoplanet Orbits for Binary Stars
Next PostNext New observations show that the Universe might not be expanding at the same rate in all directions


Since I eat out quite often I thought I'd start to post about some of the places I frequent. Angus McIndoe is located in the heart of the theater district. It's on West 44th Street between Broadway and 8th Avenue - in my opinion, the gold coast of Broadway - across the street from the Majestic Theater, the Broadhurst and the crown jewel, the Shubert Theater and Shubert Alley.
Angus McIndoe was the Eve Harrington to Joe Allen's Margo Channing. Angus was the maitre d at Joe Allen for years. Then there was the palace coup: Angus snared the prime location and, for a while, overshadowed the venerable Joe Allen. For those of you who like outside NYC, Joe Allen restaurant, on West 46th between 8th & 9th Avenues, has been the theater hang out for years. Every time I went there, I'd see at least one Broadway star - and maybe even a movie star. Joe Allen is famous for having Broadway posters adorning the walls. But the posters are of collossal flops. From Moose Murders to Carrie they were all on the wall of shame. It's a guilty pleasure to look at them and laugh.
Another great thing about having dinner at Angus - you could use the bathroom during the intermission of the show you're seeing. Even the line for the men's room can be huge at Broadway shows. With Angus you could breeze in and use the bathroom and maybe even catch a quick drink. Very civilized. I had one memorable evening at Angus. I went to see Bruce Vilanch in Hairspray. He invited me because I managed his house seats during his run. We had dinner on the top floor, which overlooks the Majestic Theater. Lucy Arnaz was there and stopped by to say hello. As did Michael Bolton. I had lunch there with Donna Murphy and Sondra Lee in its heyday. It was the first time I met Donna in person - she wanted to thank me for managing her house seats in Wonderful Town. I was trying to act nonchalant but I was a nervous wreck and starting svitzing like crazy. It was summer - I was so embarrassed. But Donna was lovely about it. She and I are both friends of Sondra - she was the original Tiger Lilly in Peter Pan and Minnie Faye in Hello Dolly. Sondra is an original member of the Actor's Studio and teaches today - I studied from her when I was pursuing acting.
But lately, the food at Angus has really gone downhill. I met a friend Richard there last week before seeing Hugh Jackman: Back on Broadway. Richard arrived first and was digging into his fried calamari. I tasted it - it reminded me of Mrs. Paul's frozen calamari - it was like lard and tasted very fishy. I had a half dozen oysters - they looked anemic and had absolutely no flavor. The place was packed as usual, but it's only going to attract unsuspecting tourists if the shoddy quality continues. This wasn't the first bad experience I've had a Angus recently. Hopefully they will turn things around. After all - I need a place to use the bathroom at intermission!
As I've posted, Vincenzino taught me some wonderful dishes when I studied with him on the Amalfi Coast. This is a favorite - zucchini with heirloom tomatoes and onions.  In the picture with me in the chef's hat on your right, I'm in Vincenzino's kitchen chopping zucchini. A lot of people comment on my huge NYC kitchen when It's actual Vincenzino's. My kitchen is your typical NYC shoebox.
I had some bottled pearl onions that I added at the last minute - they're already cooked so they heat up and dissolve quickly. Per Vincenzino - add some coarse sea salt and fresh ground pepper and nothing else. The flavors are strong enough to stand on their own. Note my recipe calls for two tablespoons of olive oil and there's obviously a good deal of liquid in the photo above. This is because I added water to speed along the process. The water mixes with the ingredients and makes a nice gravy.
Pre-heat sauté pan over medium heat.
Chop zucchini into cubes. I cut them in half lenght-wise then cut the 2 halves in half and chop into cubes. Steaming is optional but if you steam cook them for 5 - 8 minutes.
If you steam the zucchini, sauté the cherry tomatoes while they steam. If not, saute the zucchini first for a good five minutes before adding tomatoes.
Once zucchini have steamed drain and add to the pan with the tomatoes.
If desired and if zucchini are taking a bit long two cook add approximately 1/2 cup of hot water. The water will mix with the ingredients and make a nice gravy.
If you have the bottled onions, add them a minute or two before the rest of the dish is finished cooking. Add salt and pepper.
As zucchini's can vary in size you will have to do a taste test to see when they're done. Take a few cubes and let cool for a minute on a side plate and taste (you don't want to burn your mouth).
I served the zucchini with, of all things, tuna cerviche. I will post that recipe tomorrow.
There's the char we're after!
Here are my rules of thumb for cooking salmon filets: 1/2 pound serving per person and grill for two minutes on each side. As always, make sure the pan is really hot. Pre-heat over medium heat for several minutes before cooking. Put the flesh side down first - if you put the skin side down first you'll burn it off, and one of the great things about this dish is the crispy skin.
At this time of year I'm really slammed at work and I come home very tired - usually at 7:30 or 8:00 PM or even later. I want a home cooked meal but I want it to be easy. Last night I cooked the salmon and heated up brocolli rabe that I bought prepared. So, we're talking about 8 minutes of cooking/preparing start to finish. I can handle that no matter how tired I am.
1/2 lb salmon filet per person. Wild salmon is the best.
Pour 2 tbspns olive oil on the filet. The other 2 tpsns are for the skillet. My skillet is so well seasoned I don't even need to add oil.
Add the dry ingredients, rub into filet and let stand and marinate.
Pre-heat cast iron skillet over medium flame for 5 - 10 minutes. My skillet has the raised lines so you get grill marks as you would on a charcoal grill.
Grill for 2 minutes on each side - put the flesh side down first.
Even though I can be a fanatic about eating properly, I think it's important to throw caution to the wind every now and then and chow down on something that is decidely not skinny. That's what I did this weekend. Amazingly, the world didn't come to an end.
Whether you're cooking fat or skinny, presentation is almost as important as taste. And\ easy way to amp up the visuals (and taste) is to use heirloom cherry tomatoes. The flecks of color really brighten up a dish, and they're as tasty as they look. Always cut them in half. I sprinkled with coarse sea salt above.
I tried two different methods of cooking this sandwich. The first go round I cooked everything in my cast iron skillet - the sausage, tomatoes and sauerkraut. I added the roll at the end of the process.  The roll got a little extra charred and absorbed some of the liquid from the sauerkraut. It was actually delicious - I added a little Dijon mustard to the sausage and piled everything else on top.
The second time around I grilled the roll first and put it in a warm oven. I cooked the sausage and added the tomatoes, but I microwaved the saurkraut in a separate dish. This was very good, too. If you want the roll dry and crispy go with the second method. Method two will probably appeal to more of you. With the first method and you grill everything together, the roll gets a bit soggy - but I like it because it was soggy with the saurkraut liquid.
Decide for youself. Life is full of such tough decisions!
When the sausage has the grill marks on the second side remove from heat and finish off in the microwave for 45 seconds. You don't want the inside to be raw.
Note: I purchased sausage at Citarella and at Eataly. The sausage from Eataly was far superior.
Yesterday was volunteer day at the company I work for.  One day each year, all employees help out at a variety of worthy charities that run the spectrum from sewing quilts for children in cancer wards to helping out at animal shelters. We call it "Walk the Walk" day. I signed up for New York Cares. My group visited a senior center in the Projects way up in northern Manhattan. Even though I live roughly 100 - 150 blocks away, it's a completely different world up there. When I got off the subway I passed a store that sells live poultry! It was very disturbing because the poor birds were stuffed in overcrowded cages. Hens, chickens, doves, pigeons, you name it, they were selling it.
Volunteer days are as much for the volunteers as they are for the people (or animals) we're hoping to help. First, you get to know the people you work with a little bit better. Arthur and Neil work on different floors than I do and I don't get to see them very often.
Bingo is a big deal at the center. Arthur called the numbers. He had to shout because some of the residents are hard of hearing.
I left there counting my blessings. I live in my own apartment on the upper west side. I have a life outside of my apartment building. I get to go out to dinner, see Broadway shows, take spin class, go swimming and I do get that occasional trip to Europe. Sometimes it just takes a look outside your world to realize there are people who are less fortunate than you are. The people I met yesterday certainly are less fortunate, but I didn't see anyone feeling sorry for themselves. They are grateful for what they have. When you won bingo you got a t-shirt and a scratch lottery ticket. Every person who won was thrilled.


With 2010 representing the chance for a "fresh start" for so many people across the country, gyms will likely see a boost in attendance with those who have resolved to improve their health.
However, both gym rats new and old should be careful during their workouts and not over-do any exercises that could result in injuries and cause increases to health or life insurance premiums.
Megan Barclay, a physical therapist based in New York City, told the Associated Press people should be careful not to rely too heavily on ankle or knee support products to strengthen ailing muscles while working out. Unless a product is customized, most are designed for the general build of the body and may not properly adjust to specific contours of one's body.
Additionally, when exercising Barclay suggested that goals be set that can be attained and do not put to much stress on one's body.
"It is very important to set fitness goals and incorporate physical activity into your daily routine to manage stress levels," Barclay told the AP. "If time or finances prohibit you from going to the gym, find other ways to stay active such as taking a walk, running and even yard work or gardening."
For those looking for other ways to improve their health in the new decade, Barclay added that reducing chronic stress could also lead to lower blood pressure and an improved immune system.


The Palme Campsite is a small family camping site located 5 km west of Orebić near the fishing settlement of Kučište.
The floor surface of the campsite is about 1.4 hectares, covered with palm trees and other Mediterranean vegetation. There is a sandy beach sprawling along the lower edge of the campsite.
The campsite can accommodate about 150 people.
The campsite features a new sanitary block enabling a high standard of offer. It is equipped with six showers, 8 basins for laundry and dish washing, 14 washing basins, four foot wash basins, 10 toilets and 3 urinals.
The campsite also offers a fridge, laundry machine, el. fittings, and there is a possibility of filling small gas bottles. There is a general goods store in the close vicinity of the campsite (about 100 m away).


Do you ever think, I would like to experience more of the power of Jesus. I would like to live out my faith as an adventure. How do I get there? We have just the thing for you! Join us TONIGHT, from 7-9 p.m. and TOMORROW, 9:30-4:00 to learn how to experience God more fully, without being weird! Naturally. Yet, knowing that Jesus is entirely Supernatural, and you will hear how to lift your sights to His level. PAUL MACONOCHIE of 3D Movement leads us.
It's at FIRST PRESBYTERIAN CHURCH ARLINGTON, 1200 S. Collins Street, Arlington, TX. The cost is $40, which includes snacks and lunch. Just show up! No need to register. I can't wait to see you!


text:
\section{Introduction}
\label{sec:intro}

Stochastic convex optimization (SCO) is a central problem in machine
learning and statistics, where for a sample space $\S$, parameter space
$\mc{X}\subset\R^d$, and a collection of convex losses $\crl{F(\cdot;s):
  s\in\S}$, one wishes to solve
\begin{equation}\label{eq:sco}
  \minimize_{x\in\mc{X}} f(x) \defeq \E_{S\sim P}\brk*{F(x;S)}
  = \int_{\S}F(x;s)\mathrm{d}P(s)
\end{equation}
using an observed dataset $\mc{S} = S_1^n \simiid P$.  While as formulated, the
problem is by now fairly well-understood~\cite{BottouCuNo18, NemirovskiJuLaSh09,
  HazanKa11, BeckTe03, NemirovskiYu83}, it is becoming clear that, because of
considerations beyond pure statistical accuracy---memory or communication
costs~\cite{ZhangDuJoWa13_nips,GargMaNg14, BravermanGaMaNgWo16},
fairness~\cite{DworkHaPiReZe12,HashimotoSrNaLi18}, personalization or
distributed learning~\cite{McMahanMoRaHaAr17}---problem~\eqref{eq:sco} is simply
insufficient to address modern learning problems.  To that end, researchers have
revisited SCO under the additional constraint that the solution preserves the
privacy of the provided sample~\cite{DworkMcNiSm06, DworkKeMcMiNa06,
  AbadiChGoMcMiTaZh16, ChaudhuriMoSa11, DuchiJoWa13_focs}. A waypoint is
\citet{BassilySmTh14}, who provide a private method with optimal convergence
rates for the related empirical risk minimization problem, with recent papers
focus on SCO providing (worst-case) optimal rates in various settings: smooth
convex functions~\cite{BassilyFeTaTh19, FeldmanKoTa20}, non-smooth
functions~\cite{BassilyFeGuTa20}, non-Euclidean
geometry~\cite{AsiFeKoTa21,AsiDuFaJaTa21} and under more stringent privacy
constraints~\cite{LevySuAmKaKuMoSu21}.

Yet these works ground their analyses in worst-case scenarios and
provide guarantees for the \emph{hardest} instance of the class of problems they
consider. Conversely, they argue that their algorithms are optimal in a minimax
sense: for any algorithm, there exists a hard instance on which the
error achieved by the algorithm is equal to the upper bound. While valuable,
these results are pessimistic---the exhibited hard instances are typically
pathological---and fail to reflect achievable performance.

In this work, we consider the problem of adaptivity when
solving~\eqref{eq:sco} under privacy constraints. Importantly, we wish to
provide private algorithms that \emph{adapt} to the hardness of the
objective $f$. A loss function $f$ may belong to multiple problem
classes, each exhibiting different achievable rates, so a natural
desideratum is to attain the error rate of the easiest sub-class. As a
simple vignette, if one gets an arbitrary $1$-Lipschitz convex loss function
$f$, the worst-case guarantee of any $\diffp$-DP algorithm is
$\Theta(1/\sqrt{n} + d/(n\diffp))$. However, if one learns that $f$
exhibits some growth property---say $f$ is $1$-strongly convex---the
regret guarantee improves to the faster $\Theta(1/n + (d / (n\diffp))^2)$
rate with the appropriate algorithm. It is thus important to provide
algorithms that achieves the rates of the ``easiest'' class to which the
function belongs~\cite{JuditskyNe14, ZhuChDuLa16, DuchiRu21}.

To that end, consider the nested classes of functions $\mc{F}^\kappa$ for
$\kappa \in [1, \infty]$ such that, if $f \in \mc{F}^\kappa$ then there exists
$\lambda > 0$ such that for all $x \in \xdomain$,
\begin{equation*}
  %
  f(x) -
  \inf_{x'\in\mc{X}} f(x') \ge \frac{\lambda}{\kappa}\norm{x-x^\star}_2^\kappa.
\end{equation*}
For example, strong convexity implies growth with
parameter $\kappa = 2$. This growth assumption closely relates to
uniform convexity~\cite{JuditskyNe14} and the
Polyak-Kurdyka-\L{}ojasiewicz inequality~\cite{BolteNgPeSu17}, and
we make these
connections precise in Section~\ref{sec:background}. Intuitively, smaller
$\kappa$ makes the function much easier to optimize: the error around the
optimal point grows quickly. Objectives with growth are widespread in machine
learning applications: among others, the $\ell_1$-regularized hinge loss
exhibits sharp growth (i.e.\ $\kappa = 1$) while $\ell_1$- or
$\ell_\infty$-constrained $\kappa$-norm regression~---i.e.\ $s = (a, b) \in
\R^d\times\R$ and $F(x;s) = \abs{b - \tri{a, x}}^\kappa$---has
$\kappa$-growth for any $\kappa$ integer greater than
$2$~\cite{XuLiYa17}. In this work, we provide private adaptive algorithms
that adapt to the \emph{actual} growth of the function at hand.

We begin our analysis by examining Asi and Duchi's inverse sensitivity
mechanism~\cite{AsiDu20} on ERM as a motivation. While not a practical
algorithm, it achieves instance-optimal rates for any one-dimensional
function under mild assumptions, quantifying the best bound one could hope
to achieve with an adaptive algorithm, and showing (in principle) that
adaptive private algorithms can exist. We first show that for any function
with $\kappa$-growth, the inverse sensitivity mechanism achieves privacy
cost $(d/(n\diffp))^{\kappa / (\kappa - 1)}$; importantly, \emph{without
  knowledge of the function class $\mc{F}^\kappa$, that $f$ belongs
  to}. This constitutes grounding and motivation for our work in three ways:
(i) it validates our choice of sub-classes $\mc{F}^\kappa$ as the privacy
rate is effectively controlled by the value of $\kappa$, (ii) it exhibits
the rate we wish to achieve with efficient algorithms on $\mc{F}^\kappa$ and
(iii) it showcases that for easier functions,
privacy costs shrink significantly---to illustrate, for $\kappa =
5/4$ the privacy rate becomes $(d/(n\diffp))^5$.

We continue our treatment of problem~\eqref{eq:sco} under growth in
Section~\ref{sec:upper-bounds} and develop practical algorithms that achieve the
rates of the inverse sensitivity mechanism.  Moreover, for approximate
\ensuremath{(\diffp,\delta)}-differential privacy, our algorithms improve the rates, achieving roughly
$(\sqrt{d}/(n\diffp))^{\kappa / (\kappa - 1)}$.  Our algorithms hinge on a
reduction to SCO: we show that by solving a sequence of increasingly constrained
SCO problems, one achieves the right rate whenever the function exhibits growth
at the optimum. Importantly, our algorithm only requires a \emph{lower bound}
$\underline{\kappa} \le \kappa$ (where $\kappa$ is the actual growth of $f$).

We provide optimality guarantees for our algorithms in Section~\ref{sec:lb} and
show that both the inverse sensitivity and the efficient algorithms of
Section~\ref{sec:upper-bounds} are \emph{simultaneously minimax optimal} over
all classes $\mc{F}^\kappa$ whenever $\kappa = 1+\Theta(1)$ and $d=1$ for
$\diffp$-DP algorithms. Finally, we prove that in \emph{arbitrary dimension},
for both pure- and approximate-DP constraints, our algorithms are also
simultaneously optimal for all classes $\mc{F}^\kappa$ with $\kappa \ge 2$.

On the way, we provide results that may be of independent
interest to the community. First, we develop optimal algorithms for SCO
under \emph{pure} differential privacy constraints, which, to the best of
our knowledge, do not exist in the literature. Secondly, our algorithms and
analysis provide high-probability bounds on the loss, whereas existing
results only provide (weaker) bounds on the expected loss.  Finally, we
complete the results of~\citet{RamdasSi13} on (non-private)
optimization lower bounds for functions with $\kappa$-growth by providing
information-theoretic lower bounds (in contrast to oracle-based
lower bounds that rely on observing only gradient information) and
capturing the optimal dependence on all problem parameters (namely $d, \lip$
and $\lambda$).

\subsection{Related work}
\label{sec:related-work}

Convex optimization is one of the best studied problems in private data
analysis~\cite{ChaudhuriMoSa11,DuchiJoWa13_focs,SmithTh13lasso,BassilySmTh14}.
The first papers in this line of work mainly study minimizing the empirical
loss, and readily establish that the (minimax) optimal privacy rates are
${d}/{n \diffp}$ for pure $\diffp$-DP and ${\sqrt{d \log(1/\delta)}}/{n \diffp}$
for $(\diffp,\delta)$-DP~\cite{ChaudhuriMoSa11,BassilySmTh14}. More recently,
several works instead consider the harder problem of privately minimizing the
population loss~\cite{BassilyFeTaTh19,FeldmanKoTa20}. These papers introduce new
algorithmic techniques to obtain the worst-case optimal rates of
$1/\sqrt{n}+ {\sqrt{d \log(1/\delta)}}/{n \diffp}$ for \ensuremath{(\diffp,\delta)}-DP. They also show
how to improve this rate to the faster
$1/n + {{d \log(1/\delta)}}/{(n \diffp)^2}$ in the case of $1$-strongly convex
functions. Our work subsumes both of these results as they correspond to
$\kappa = \infty$ and $\kappa = 2$ respectively. To the best of our knowledge,
there has been no work in private optimization that investigates the rates under
general $\kappa$-growth assumptions or adaptivity to such conditions.

In contrast, the optimization community has extensively studied growth assumptions
~\cite{RamdasSi13,JuditskyNe14,ChatterjeeDuLaZh16} and show that on these
problems, carefully crafted algorithms improves upon the standard $1/\sqrt{n}$
for convex functions to the faster $(1/\sqrt{n})^{\kappa/(\kappa -
	1)}$. \cite{JuditskyNe14} derives worst-case optimal (in the first-order
oracle model) gradient algorithms in the uniformly convex case (i.e.
$\kappa \ge 2$) and provides technique to adapt to the growth $\kappa$, while
\cite{RamdasSi13}, drawing connections between growth conditions and active
learning, provides upper and lower bounds in the first-order stochastic oracle
model. We complete the results of the latter and provide
\emph{information-theoretic} lower bounds that have optimal dependence on
$d, \lambda$ and $n$---their lower bound only holding for $\lambda$ inversely
proportional to $d^{1/2-1/\kappa}$, when $\kappa \ge 2$. Closest to our work
is~\cite{ChatterjeeDuLaZh16} who studies instance-optimality via local minimax
complexity~\cite{CaiLo15}. For one-dimensional functions, they develop a
bisection-based instance-optimal algorithm and show that on individual functions
of the form $t \mapsto \kappa^{-1}\abs{t}^\kappa$, the local minimax rate is
$(1/\sqrt{n})^{\kappa / (\kappa - 1)}$.

\begin{comment}
Recently, there has been works that study adaptivity to the difficulty of the
underlying instance with differential privacy~\cite{AsiDu20,AsiDu20nips}. These
works develop an inverse sensitivity mechanism and show that it enjoys
instance-optimality guarantees for general functions.  However,
\citet{AsiDu20,AsiDu20nips} do not study the problem of interest in this paper,
that is, private optimization with growth conditions. Their techniques, though,
will prove useful in our setting as we show in~\Cref{sec:inv-sens}.\dl{this
reads a little weird; if it's instance-optimal, it should still be optimal for
our growth conditions?}
\end{comment}

\section{Preliminaries}
\label{sec:background}

We first provide notation that we use throughout this paper, define useful
assumptions and present key definitions in convex analysis and differential
privacy.

\paragraph{Notation.} $n$ typically denotes the sample size and $d$ the
dimension. Throughout this work, $x$ refers to the optimization variable,
$\mc{X} \subset \R^d$ to the constraint set and $s$ to elements ($S$ when random) of
the sample space $\S$. We usually denote by $F:\mc{X}\times\S\to\R$ the
(convex) loss function and for a dataset
$\mc{S} = (s_1, \ldots, s_n) \subset \S$, we define the empirical and population
losses
\begin{equation*}
  \femp(x) \defeq \frac{1}{n}\sum_{i\le n}F(x;s_i) \mbox{~~and~~} f(x) \defeq \E_{S\sim P}\brk*{F(x;S)}.
\end{equation*}
We omit the dependence on $P$ as it is often clear from context. We reserve
$\diffp, \delta \ge 0$ for the privacy parameters of Definition~\ref{def:DP}. We
always take gradients with respect to the optimization variable $x$. In the case
that $F(\cdot;s)$ is not differentiable at $x$, we override notation and define
$\nabla F(x;s) = \argmin_{g\in \partial F(x;s)}\norm{g}_2$, where
$\partial F(x;s)$ is the subdifferential of $F(\cdot;s)$ at $x$. We use
$\msf{A}$ for (potentially random) mechanism and $S_1^n$ as a shorthand for
$\prn*{S_1, \ldots, S_n}$. For $p \ge 1$, $\norm{\cdot}_p$ is the standard
$\ell_p$-norm, $\ball_p^d(R)$ is the corresponding
$d$-dimensional $p$-ball of radius $R$ and $p^\star$ is the dual of $p$, i.e.\
such that $1/p^\star + 1/p = 1$. Finally, we define the Hamming distance between
datasets
$\dham(\mc{S}, \mc{S}') \defeq \inf_{\sigma \in \mathfrak{S}_n} \ones\crl{s_i
  \neq s'_{\sigma(i)}}$, where $\mathfrak{S}_n$ is the set of permutations over
sets of size $n$.

\paragraph{Assumptions.} We first state standard assumptions for
solving~\eqref{eq:sco}. We assume that $\mc{X}$ is a closed, convex domain such
that $\diam_2(\mc{X}) = \sup_{x, y \in \mc{X}}\norm{x-y}_2 \le \rad <
\infty$. Furthermore, we assume that for any $s\in\S$, $F(\cdot;s)$ is convex
and $\lip$-Lipschitz with respect to $\norm{\cdot}_2$. Central to our work, we
define the following $\kappa$-growth assumption.

\begin{assumption}[$\kappa$-growth]\label{ass:growth}
  Let $x\opt = \argmin_{x\in\xdomain}\ff(x)$. For a loss $F$ and distribution
  $P$, we say that $(F, P)$ has $(\lambda, \kappa)$ growth for
  $\kappa\in[1, \infty]$ and $\lambda > 0$, if the population function satisfies
  \begin{equation*}
    \mbox{for all~} x\in\mc{X}, \quad f(x) - f(x^\star) \ge \frac{\lambda}{\kappa} \ltwo{x-x^\star}^\kappa.
  \end{equation*}
  In the case where $\what{P}$ is the empirical distribution on a finite dataset
  $\mc{S}$, we refer to $(\lambda, \kappa)$-growth of $(F, \what{P})$ as
  $\kappa$-growth of the empirical function $\femp$.
\end{assumption}

\paragraph{Uniform convexity and \klinequality{} inequality.}
Assumption~\ref{ass:growth} is closely related to two fundamental notions in
convex analysis: uniform convexity and the \klinequality{}
inequality. Following~\cite{Nesterov08}, we say that
$h:\mc{Z}\subset\R^d \to \R$ is $(\sigma, \kappa)$-uniformly convex with
$\sigma > 0$ and $\kappa \ge 2$ if
\begin{equation*}
  \mbox{for all~~}x,y \in\mc{Z}, \quad h(y) \ge h(x) + \tri{\nabla h(x), y-x}
  + \frac{\sigma}{\kappa}\norm{x-y}_2^\kappa.
\end{equation*}
This immediately implies that (i) sums (and expectations) preserve uniform
convexity (ii) if $f$ is uniformly convex with $\lambda$ and $\kappa$, then it
has $(\lambda, \kappa)$-growth. This will be useful when constructing hard
instances as it will suffice to consider $(\lambda, \kappa)$-uniformly convex
functions which are generally more convenient to manipulate. Finally, we point
out that, in the general case that $\kappa \ge 1$, the literature refers to
Assumption~\ref{ass:growth} as the \klinequality{}
inequality~\cite{BolteNgPeSu17} with, in their notation,
$\varphi(s) = (\kappa / \lambda)^{1/\kappa}s^{1/\kappa}$. Theorem 5-(ii)
in~\cite{BolteNgPeSu17} says that, under mild conditions,
Assumption~\ref{ass:growth} implies the following inequality between the error
and the gradient norm for all $x \in \xdomain$
\begin{equation}\label{eq:kl}
  %
  f(x) - \inf_{x'\in\mc{X}} f(x')
  %
  %
  \le
  \frac{e}{\lambdae}\nrm*{\nabla f(x)}_2^{\tfrac{\kappa}{\kappa - 1}},
\end{equation}
This is a key result in our analysis of the inverse sensitivity mechanism of
Section~\ref{sec:inv-sens}.

\paragraph{Differential privacy.} We begin by recalling the definition of
\ensuremath{(\diffp,\delta)}-differential privacy.
\begin{definition}[\cite{DworkMcNiSm06,DworkKeMcMiNa06}]
	\label{def:DP}
	A randomized  algorithm $\mathsf{A}$ is \ensuremath{(\diffp,\delta)}-differentially private (\ensuremath{(\diffp,\delta)}-DP) if, for all datasets $\Ds,\Ds' \in \domain^n$ that differ in a single data element and for all events $\mathcal{O}$ in the output space of $\mathsf{A}$, we have
	\[
	\Pr \left( \mathsf{A}(\Ds)\in \mathcal{O} \right) \leq e^{\eps} \Pr \left(\mathsf{A}(\Ds')\in \mathcal{O} \right) +\delta.
	\]
\end{definition}

We use the following standard results in differential privacy.
\begin{lemma}[Composition~{\cite[][Thm.~3.16]{DworkRo14}}]
	\label{lemma:basic-comp}
	If $\mathsf{A}_1,\dots,\mathsf{A}_k$ are randomized algorithms that each is
        $(\diffp, \delta)$-DP, then their composition
        $(\mathsf{A}_1(\Ds),\dots,\mathsf{A}_k(\Ds))$ is $(k \diffp, k\delta)$-DP.
\end{lemma}

Next, we consider the Laplace mechanism. We will let $Z \sim \laplace_d(\sigma)$ denote a $d$-dimensional vector $Z \in \R^d$ such that $Z_i \simiid \laplace(\sigma)$ for $1 \le i \le d$.
\begin{lemma}[Laplace mechanism~{\cite[][Thm.~3.6]{DworkRo14}}]
	\label{lemma:lap-mech}
	Let $h: \domain^n \to \R^d$ have $\ell_1$-sensitivity $\Delta$,
	that is, $\sup_{\Ds,\Ds' \in \domain^n: \dham(\Ds,\Ds') \le 1} \lone{h(\Ds) - h(\Ds')} \le \Delta$. Then the Laplace mechanism $\mathsf{A}(\Ds) = h(\Ds) + \laplace_d(\sigma)$ with $\sigma = \Delta/\diffp$ is $\diffp$-DP.
\end{lemma}

Finally, we need the Gaussian mechanism for \ensuremath{(\diffp,\delta)}-DP.
\begin{lemma}[Gaussian mechanism~{\cite[][Thm.~A.1]{DworkRo14}}]
	\label{lemma:gauss-mech}
	Let $h: \domain^n \to \R^d$ have $\ell_2$-sensitivity $\Delta$,
	that is, $\sup_{\Ds,\Ds' \in \domain^n: \dham(\Ds,\Ds') \le 1} \ltwo{h(\Ds) - h(\Ds')} \le \Delta$. Then the Gaussian mechanism $\mathsf{A}(\Ds) = h(\Ds) + \normal(0,\sigma^2 I_d)$ with $\sigma = 2 \Delta \log(2/\delta)/\diffp$ is $(\diffp,\delta)$-DP.
\end{lemma}

\paragraph{Inverse sensitivity mechanism.} Our goal is to design private
optimization algorithms that adapt to the difficulty of the underlying
function. As a reference point, we turn to the inverse sensitivity mechanism
of~\cite{AsiDu20} as it enjoys general instance-optimality guarantees.  For a
given function $h: \domain^n \to \mathcal{T} \subset \R^d$ that we wish to
estimate privately, define the \emph{inverse sensitivity} at $x\in\mc{T}$
\begin{equation}
\label{eq:inv-sens}
	\invmodcont_h(\Ds;x) = \inf_{\Ds'} \{ \dham(\Ds',\Ds) : h(\Ds')=x  \},
\end{equation}
that is, the inverse sensitivity of a target parameter $y \in \mathcal{T}$ at
instance $\Ds$ is the minimal number of samples one needs to change to reach a
new instance $\Ds'$ such that $h(\Ds') = y$. Having this quantity, the inverse
sensitivity mechanism samples an output from the following probability density
\begin{equation}
\label{eq:inv-sens}
\pdf_{\Ainvsm(\Ds)} (x) \propto e^{- \diffp \invmodcont_h(\Ds;x) }.
\end{equation}
The inverse sensitivity mechanism preserves $\diffp$-DP and enjoys instance-optimality guarantees in general settings~\cite{AsiDu20}. In contrast to (worst-case) minimax optimality guarantees which measure the performance of the algorithm on the hardest instance, these notions of instance-optimality provide stronger per-instance optimality guarantees.

\section{Adaptive rates through inverse sensitivity for $\diffp$-DP}
\label{sec:inv-sens}

To understand the achievable rates when privately optimizing functions with
growth, we begin our theoretical investigation by examining the inverse
sensitivity mechanism in our setting.
We show that, for instances that exhibit $\kappa$-growth of the
empirical function, the inverse sensitivity mechanism privately solves ERM with
excess loss roughly $({d}/{n \diffp} )^{\frac{\kappa}{\kappa-1}}$.

In our setting, we use a gradient-based approximation of the inverse sensitivity mechanism to simplify the analysis, while attaining similar rates. Following~\cite{AsiDu20nips} with our function of interest $h(\Ds) \defeq \argmin_{x \in \xdomain} \ff_\Ds(x)$, we can lower bound the inverse sensitivity $\invmodcont_h(\Ds;x) \ge n \ltwo{\nabla f_\Ds(x)}/2 \lip$ under natural assumptions. We define a $\rho$-smoothed version of this quantity which is more suitable to continuous domains
\begin{equation*}
G^\rho_\Ds(x) = \inf_{y \in \xdomain : \ltwo{y-x} \le \rho} \ltwo{\nabla f_\Ds(y)},
\end{equation*}
and define the $\rho$-smooth gradient-based inverse sensitivity mechanism
\begin{equation}
\label{eq:grad-inv-sens}
\pdf_{\Agrinvsm(\Ds)} (x) \propto e^{- \diffp n G^\rho_\Ds(x)/2 \lip }.
\end{equation}

Note that while exactly sampling from the un-normalized density
$\pdf_{\Agrinvsm\prn{\Ds}}$ is computationally intractable, analyzing its
performance is an important step towards understanding the optimal rates for the
family of functions with growth that we study in this work. The following
theorem demonstrates the adaptivity of the inverse sensitivity mechanism to the
growth of the underlying instance. We defer the proof
to~\Cref{sec:apdx-inv-sens}.
\begin{restatable}{theorem}{restateInvSens}
	\label{thm:inv-sens}
	Let $\Ds=(\ds_1, \ldots, \ds_n)\in \domain^n$, $F(x;\ds)$ be convex, $\lip$-Lipschitz for all $\ds \in \domain$.
	Let $x\opt = \argmin_{x \in \xdomain} \ff_\Ds(x)$ and assume $x\opt$ is in the interior of $\xdomain$. Assume that $\ff_S(x)$ has $\kappa$-growth (Assumption~\ref{ass:growth}) with $\kappa \ge \lkappa > 1$.
	For $\rho>0$, the $\rho$-smooth inverse sensitivity mechanism $\Agrinvsm$~\eqref{eq:grad-inv-sens} is $\diffp$-DP, and with probability at least $1-\beta$
	the output $\hat x = \Agrinvsm(\Ds) $ has
	\begin{equation*}
	 \ff_\Ds(\hat x) - \min_{x \in \xdomain} \ff_\Ds(x)
	\le  \frac{1}{\lambda^{\frac{1}{\kappa-1}}} \left(\frac{2 \lip (\log(1/\beta) + d \log(\rad/\rho))}{n \diffp} \right)^{\frac{\kappa}{\kappa-1}} + \lip \rho.
	\end{equation*}
	Moreover, setting $\rho = ({\lip}/{\sc})^{\frac{1}{\lkappa-1}} ({d}/{n \diffp})^{\frac{\lkappa}{\lkappa-1}}$, we have
	\begin{align*}
	\ff_\Ds(\hat x) - \min_{x \in \xdomain} \ff_\Ds(x)
	%
	& \le  \frac{1}{\lambda^{\frac{1}{\kappa-1}}} \wt O \left( \frac{\lip d}{n \diffp} \right)^{\frac{\kappa}{\kappa-1}}.
	\end{align*}
\end{restatable}

The rates of the inverse sensitivity in~\Cref{thm:inv-sens} provide two main
insights regarding the landscape of the problem with growth conditions. First,
these conditions allow to improve the worst-case rate $d/n\diffp$ to
$({d}/{n \diffp} )^{\frac{\kappa}{\kappa-1}}$ for pure $\diffp$-DP and therefore
suggest a better rate
$({\sqrt{d \log(1/\delta)}}/{n \diffp} )^{\frac{\kappa}{\kappa-1}}$ is possible
for approximate \ensuremath{(\diffp,\delta)}-DP. Moreover, the general instance-optimality guarantees of
this mechanism~\cite{AsiDu20} hint that these are the optimal rates for our
class of functions. In the sections to come, we validate the correctness of
these predictions by developing efficient algorithms that achieve these rates
(for pure and approximate privacy), and prove matching lower bounds which
demonstrate the optimality of these algorithms.

\section{Efficient algorithms with optimal rates}
\label{sec:upper-bounds}

While the previous section demonstrates that there exists algorithms that
improve the rates for functions with growth, we pointed out that $\Agrinvsm$ was
computationally intractable in the general case. In this section, we develop
efficient algorithms---e.g.\ that are implementable with gradient-based
methods---that achieve the same convergence rates. Our algorithms build on the
recent localization techniques that~\citet{FeldmanKoTa20} used to obtain optimal
rates for DP-SCO with general convex functions. In Section~\ref{sec:dp-sco-hb},
we use these techniques to develop private algorithms that achieve the optimal
rates for (pure) DP-SCO with high probability, in contrast to existing results
which bound the expected excess loss. These results are of independent interest.

In Section~\ref{sec:dp-sco-growth}, we translate these results into convergence
guarantees on privately optimizing convex functions with growth by solving a
sequence of increasingly constrained SCO problems---the high-probability
guarantees of~\Cref{sec:dp-sco-hb} being crucial to our convergence analysis of
these algorithms.

\subsection{High-probability guarantees for convex DP-SCO}
\label{sec:dp-sco-hb}

We first describe our algorithm (Algorithm~\ref{alg:pure-erm}) then analyze its
performance under pure-DP (Proposition~\ref{thm:sco-hb-pure}) and approximate-DP
constraints (Proposition~\ref{thm:sco-hb-appr}). Our analysis builds on novel
tight generalization bounds for uniformly-stable algorithms with high
probability~\cite{FeldmanVo19}. We defer the proofs
to~\Cref{sec:apdx-sec:upper-bounds}.

\begin{algorithm}
	\caption{Localization-based Algorithm}
	\label{alg:pure-erm}
	\begin{algorithmic}[1]
		\REQUIRE Dataset $\Ds=(\ds_1, \ldots, \ds_n)\in \domain^n$,
		constraint set $\xdomain$,
		step size $\ss$, initial point $x_0$,
		privacy parameters $(\diffp,\delta)$;
		\STATE Set $k = \ceil{\log n}$ and $n_0 = n/k$
		\FOR{$i=1$ to $k$\,}
		\STATE Set
		$\ss_i = 2^{-4i} \ss$
		\STATE Solve the following ERM over
		$ \mc{X}_i= \{x\in \xdomain: \norm{x - x_{i-1}}_2 \le {2\lip \ss_i n_0} \}$:\\
		$%
		\displaystyle
		\quad \quad \quad \quad
		F_i(x) =  \frac{1}{n_0} \sum_{j=1 + (i-1)n_0}^{in_0} F(x;\ds_j) + \frac{1}{\ss_i n_0 } \norm{x - x_{i-1}}_2^2
		$%
		\STATE Let $\hat x_i$ be the output of the optimization algorithm
		\IF{$\delta=0$}
		\STATE Set
		$\noise_i \sim \laplace_d(\sigma_i)$ where $\sigma_i = 4 \lip \ss_i \sqrt{d}/\diffp_i$
		\ELSIF{$\delta>0$}
		\STATE Set $\noise_i \sim \normal(0,\sigma_i^2)$ where $\sigma_i = 4 \lip \ss_i \sqrt{\log(1/\delta)}/\diffp$
		\ENDIF
		\STATE Set $x_i = \hat x_i + \noise_i$
		\ENDFOR
		\RETURN the final iterate $x_k$
	\end{algorithmic}
\end{algorithm}

\begin{restatable}{proposition}{restatePureSCOHb}
 	\label{thm:sco-hb-pure}
 	Let $\beta \le 1/(n+d)$, $\diam_2(\xdomain)\le \rad$ and $F(x;\ds)$ be convex, $\lip$-Lipschitz for all $\ds \in \domain$. Setting
 	\begin{equation*}
 	\ss = \frac{\rad}{\lip} \min \left(\frac{1}{\sqrt{n \log(1/\beta)}} ,\frac{ \diffp}{ d \log(1/\beta)} \right)
 	\end{equation*}
 	then for $\delta=0$, \Cref{alg:pure-erm} is $\diffp$-DP and has with probability $1-\beta$
 	\begin{equation*}
 	f(x) -  f(x\opt)
 	\le \lip \rad \cdot  O \left( \frac{\sqrt{\log(1/\beta) } \log^{3/2} n}{\sqrt{n}} + \frac{ d \log(1/\beta) \log n}{n \diffp} \right).
 	%
 	\end{equation*}
\end{restatable}

Similarly, by using a different choice for the parameters and noise distribution, we have the following guarantees for approximate \ensuremath{(\diffp,\delta)}-DP.
 \begin{restatable}{proposition}{restateApproxSCOHb}
 	\label{thm:sco-hb-appr}
 	Let $\beta \le 1/(n+d)$, $\diam_2(\xdomain)\le \rad$ and $F(x;\ds)$ be convex, $\lip$-Lipschitz for all $\ds \in \domain$. Setting
 	\begin{equation*}
 	\ss = \frac{\rad}{\lip} \min \left(\frac{1}{\sqrt{n \log(1/\beta)}} ,\frac{ \diffp}{ \sqrt{d \log(1/\delta)} \log(1/\beta)} \right),
 	\end{equation*}
 	then for $\delta > 0 $, \Cref{alg:pure-erm} is $(\diffp,\delta)$-DP and has with probability $1-\beta$
 	\begin{equation*}
 	f(x) -  f(x\opt)
 	\le \lip \rad \cdot  O \left( \frac{\sqrt{\log(1/\beta) } \log^{3/2} n}{\sqrt{n}} + \frac{ \sqrt{d \log(1/\delta)} \log(1/\beta) \log n}{n \diffp} \right).
 	%
 	\end{equation*}
 \end{restatable}

\subsection{Algorithms for DP-SCO with growth}
\label{sec:dp-sco-growth}
Building on the algorithms of the previous section, we design algorithms that
recover the rates of the inverse sensitivity mechanism for functions with
growth, importantly without knowledge of the value of $\kappa$. Inspired by
epoch-based algorithms from the optimization
literature~\cite{JuditskyNe10,HazanKa11}, our algorithm iteratively applies the
private procedures from the previous section. Crucially, the growth assumption
allows to reduce the diameter of the domain after each run, hence improving the
overall excess loss by carefully choosing the hyper-parameters. We provide full
details in~\Cref{alg:loc-growth}.

\begin{algorithm}
	\caption{Epoch-based algorithms for $\kappa$-growth}
	\label{alg:loc-growth}
	\begin{algorithmic}[1]
		\REQUIRE
		Dataset $\Ds=(\ds_1, \ldots, \ds_n)\in \domain^n$,
		convex set $\xdomain$,
		initial point $x_0$,
		number of iterations $T$,
		%
		privacy parameters $(\diffp,\delta)$;
		\STATE Set $n_0 = n/T$ and $\rad_0 = \diam_2(\xdomain)$
		\IF{$\delta=0$}
		\STATE Set
		$\ss_0 = \frac{\rad_0}{2\lip} \min \left(\frac{1}{\sqrt{n_0 \log(n_0) \log(1/\beta)}} ,\frac{ \diffp}{ d \log(1/\beta)} \right)$
		\ELSIF{$\delta>0$}
		\STATE $\ss_0 = \frac{\rad_0}{2\lip} \min \left(\frac{1}{\sqrt{n_0 \log(n_0) \log(1/\beta)}} ,\frac{ \diffp}{ \sqrt{d \log(1/\delta)} \log(1/\beta)} \right)$
		\ENDIF
		\FOR{$i=0$ to $T - 1$\,}
		\STATE Let $\Ds_i = (\ds_{1+(i-1) n_0}, \dots, \ds_{i n_0})$
		\STATE Set $\rad_i = 2^{-i} \rad_0$ and $\ss_i = 2^{-i} \ss_0$
		\STATE Set $\xdomain_i = \crl{x \in \xdomain : \ltwo{x - x_i} \le \rad_i}$
		\STATE Run~\Cref{alg:pure-erm} on dataset $\Ds_i$ with starting point $x_i$, privacy parameter $(\diffp,\delta)$, domain $\xdomain_i$ (with diameter $\rad_i$), step size $\ss_i$
		\STATE Let $x_{i+1}$ be the output of the private procedure
		\ENDFOR
		\RETURN $x_T$
	\end{algorithmic}
	\label{Alg:pure-with-growth}
\end{algorithm}

The following theorem summarizes our main upper bound for DP-SCO with growth in the pure privacy model, recovering the rates of the inverse sensitivity mechanism in~\Cref{sec:inv-sens}. We defer the proof to~\Cref{sec:proof-thm-sco-growth-pure}.
\begin{restatable}{theorem}{restateSCOGrowthPure}
	\label{thm:sco-growth-pure}
	Let $\beta \le 1/(n+d)$, $\diam_2(\xdomain)\le \rad$ and $F(x;\ds)$ be convex, $\lip$-Lipschitz for all $\ds \in \domain$.
	Assume that $\ff$ has $\kappa$-growth (Assumption~\ref{ass:growth}) with $\kappa \ge \lkappa >1$.
	Setting $T = \left\lceil \frac{2 \log n}{\lkappa-1} \right\rceil$,
	\Cref{alg:loc-growth} is $\diffp$-DP and has with probability $1-\beta$
	\begin{equation*}
	\ff(x_T) -  \min_{x \in \xdomain} \ff(x)
	\le  \frac{1}{\lambda^{\frac{1}{\kappa - 1}}} \cdot  \wt O \left( \frac{\lip \sqrt{\log(1/\beta) } }{\sqrt{n}} + \frac{ \lip d \log(1/\beta)}{n \diffp (\lkappa -1)} \right)^{\frac{\kappa}{\kappa-1}},
	\end{equation*}
	where $\wt O$ hides logarithmic factors depending on $n$ and $d$.
\end{restatable}

\begin{proof}[Sketch of the proof]
  The main challenge of the proof is showing that the iterate achieves good risk
  without knowledge of $\kappa$. Let us denote by $D\cdot\rho$ the error
  guarantee of Proposition~\ref{thm:sco-hb-pure} (or
  Proposition~\ref{thm:sco-hb-appr} for approximate-DP). At each stage $i$, as
  long as $x^\star = \argmin_{x\in\mc{X}} f(x)$ belongs to $\mc{X}_i$, the
  excess loss is of order $D_i\cdot\rho$ and thus decreases exponentially fast
  with $i$. The challenge is that, without knowledge of $\kappa$, we do not know
  the index $i_0$ (roughly $\tfrac{\log_2 n}{\kappa - 1}$) after which
  $x^\star \notin D_j$ for $j\ge i_0$ and the regret guarantees become
  meaningless with respect to the original problem. However, in the stages after
  $i_0$, as the constraint set becomes very small, we upper bound the variations
  in function values $f(x_{j+1}) - f(x_j)$ and show that the sub-optimality
  cannot increase (overall) by more than $O(D_{i_0}\cdot\rho)$, thus achieving
  the optimal rate of stage $i_0$.

  %
  %
  %
  %
  %
  %
  %
  %
  %
  %
  %
  %
  %
\end{proof}

Moreover, we can improve the dependence on the dimension for approximate \ensuremath{(\diffp,\delta)}-DP, resulting in the following bounds.

\begin{restatable}{theorem}{restateSCOGrowthApprox}
	\label{thm:sco-growth-appr}
	Let $\beta \le 1/(n+d)$, $\diam_2(\xdomain)\le \rad$ and $F(x;\ds)$ be convex, $\lip$-Lipschitz for all $\ds \in \domain$.
	Assume that $\ff$ has $\kappa$-growth (Assumption~\ref{ass:growth}) with $\kappa \ge \lkappa >1$.
	Setting $T = \left\lceil \frac{2 \log n}{\lkappa-1} \right\rceil$ and $\delta>0$, \Cref{alg:loc-growth} is $(\diffp,\delta)$-DP and has with probability $1-\beta$
	\begin{equation*}
	\ff(x_T) -  \min_{x \in \xdomain} \ff(x)
	\le \frac{1}{\lambda^{\frac{1}{\kappa - 1}}} \cdot \wt O \left( \frac{\lip \sqrt{\log(1/\beta) } }{\sqrt{n}} + \frac{ \lip \sqrt{d \log(1/\delta)} \log(1/\beta)}{n \diffp (\lkappa -1)} \right)^{\frac{\kappa}{\kappa-1}},
	\end{equation*}
	where $\wt O$ hides logarithmic factors depending on $n$ and $d$.
\end{restatable}

\section{Lower bounds}\label{sec:lb}

In this section, we develop (minimax) lower bounds for the problem of SCO with
$\kappa$-growth under privacy constraints. Note that taking $\diffp \to \infty$
provides lower bound for the unconstrained minimax risk. For a sample space
$\S$ and collection of distributions $\mc{P}$ over $\S$, we define the function
class $\mc{F}^\kappa(\mc{P})$ as the set of convex functions from $\R^d\to\R$
that are $\lip$-Lipschitz and has $\kappa$-growth
(Assumption~\ref{ass:growth}).
We define the \emph{constrained} minimax risk~\cite{BarberDu14a}
\begin{equation}
  \mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \diffp, \delta) \defeq \inf_{\what{x}_n\in\mc{A}^{\diffp, \delta}}
  \sup_{(F, P) \in \mc{F}^\kappa\times\mc{P}} \E\brk*{
    f(\what{x}_n(S_1^n)) - \inf_{x'\in\mc{X}}f(x')
  },
\end{equation}
where $\mc{A}^{\epsilon, \delta}$ is the collection of $(\diffp, \delta)$-DP
mechanisms from $\S^n$ to $\mc{X}$. When clear from context, we omit the
dependency on $\mc{P}$ of the function class and simply write
$\mc{F}^\kappa$. We also forego the dependence on $\delta$ when referring to
pure-DP constraints, i.e.
$\mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \diffp, \delta=0) \eqdef
\mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \diffp)$. We now proceed to prove
tight lower bounds for $\diffp$-DP in~\Cref{sec:lb-pure} and \ensuremath{(\diffp,\delta)}-DP
in~\Cref{sec:lb-appr}.

\subsection{Lower bounds for pure $\diffp$-DP}\label{sec:lb-pure}

Although in~\Cref{sec:upper-bounds} we show that the same algorithm achieves the
optimal upper bounds for all values of $\kappa >1$, the landscape of the problem
is more subtle for the lower bounds and we need to delineate two different cases
to obtain tight lower bounds. We begin with $\kappa \ge 2$,
which %
corresponds to uniform convexity and enjoys properties that make the problem
easier (e.g., closure under summation or addition of linear terms).  The second
case, $1 < \kappa <2$, corresponds to sharper growth and requires a different
hard instance to satisfy the growth
condition.%

\paragraph{$\kappa$-growth with $\kappa \ge 2$. } We begin by developing
lower bounds under pure DP for $\kappa \ge 2$ %

\begin{restatable}[Lower bound for $\diffp$-DP, $\kappa\ge 2$]
  {theorem}{restateLbPrivateLargeKappa}
  \label{thm:lb-sco-private-large-kappa}
    Let $d \ge 1$, $\mc{X} = \ball_2^d(R)$, $\S = \crl{\pm e_j}_{j\le d}$,
  $\kappa \ge 2$ and $n\in\N$. Let $\mc{P}$ be the set of distributions on
  $\S$. Assume that
  %
  %
  %
  %
  \begin{equation*}
    2^{\kappa-1}
    \le
    \frac{\lip}{\sc} \frac{1}{ R^{\kappa - 1}}
    \le   2^{\kappa-1}
    \sqrt{96 n} \mbox{~~and~~} n\diffp \ge \frac{1}{\sqrt{3}}
  \end{equation*}

  The following lower bound holds
  \begin{equation}\label{eq:lb-private-large-kappa}
	\mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \epsilon) \ge
	\frac{1}{\lambda^{\tfrac{1}{\kappa - 1}}}\tilde{\Omega}\prn*{
	\prn*{\frac{L}{\sqrt{n}}}^{\tfrac{\kappa}{(\kappa-1)}}
	+
	\prn*{\frac{Ld}{n\diffp}}^{\tfrac{\kappa}{\kappa - 1}}
}.
	\end{equation}
\end{restatable}

First of all, note that $L \ge \lambda 2^\kappa R^{\kappa - 1}$ is not an
overly-restrictive assumption. Indeed, for an arbitrary
$(\lambda, \kappa)$-uniformly convex and $\lip$-Lipschitz function, it always
holds that $L \ge \tfrac{\lambda}{2}R^{\kappa - 1}$. This is thus equivalent to
assuming $\kappa = \Theta(1)$. Note that when $\kappa \gg 1$, the standard
$n^{-1/2} + d/(n\diffp)$ lower bound holds. We present the proof
in~\Cref{app:lb-pure-large-kappa} and preview the main ideas here.
\begin{proof}[Sketch of the proof]
  Our lower bounds hinges on the collections of functions
  $F(x;s) \defeq a\kappa^{-1}\norm{x}_2^\kappa + b\tri{x, s}$ for $a, b\ge 0$ to
  be chosen later. These functions are~\cite[][Lemma 4]{Nesterov08}
  $\kappa$-uniformly convex for any $s\in\S$ and in turn, so is the population
  function $f$. We proceed as follows, we first prove an information-theoretic
  (non-private) lower bound (\Cref{thm:lb-non-private-large-kappa} in
  Appendix~\ref{app:lb-pure-large-kappa}) which provides the statistical term
  in~\eqref{eq:lb-private-large-kappa}. With the same family of functions, we
  exhibit a collection of datasets and prove by contradiction that if an
  estimator were to optimize below a certain error it would have violated
  $\diffp$-DP---this yields a lower bound on ERM for our function class
  (Theorem~\ref{thm:lb-private-large-kappa} in
  Appendix~\ref{app:lb-pure-large-kappa}). We conclude by proving a reduction
  from SCO to ERM in~\Cref{sec:lb-erm-sco}.
\end{proof}

\paragraph{$\kappa$-growth with $\kappa \in (1, 2]$.}
As the construction of the hard instance is more intricate for
$\kappa < 2$, we provide a one-dimensional lower bound and leave the
high-dimensional case for future work.
In this case we directly obtain the result with a
private version of Le Cam's method~\cite{Yu97,Wainwright19,BarberDu14a}, however
with a different family of functions.

The issue with the construction of the previous section is that the function
does not exhibit sharp growth for $\kappa < 2$. Indeed, the added linear
function shifts the minimum away from $0$ where the function is differentiable
and as a result it locally behaves as a quadratic and only achieves growth
$\kappa = 2$. To establish the lower bound, we consider a different sample
function $F$ that has growth exactly $1$ on one side and $\kappa$ on the other
side. This yields the following

\begin{restatable}[Lower bound for $\diffp$-DP, $\kappa \in {(1, 2]}$
	%
	]{theorem}{restateLbSmallKappa}\label{thm:lb-small-kappa}
   Let $d=1$, $\S = \crl{-1, +1}$, $\kappa \in (1, 2]$, $\lambda=1$, $\lip=2$, and $n\in\N$. There exists a collection
  of distributions $\mc{P}$ such that, whenever $n\diffp \ge 1/\sqrt{3}$, it holds that
  %
  %
  %
  %
  %
  %
  \begin{equation}
	\mathfrak{M}_n(\brk{-1, 1}, \mc{P}, \mc{F}^\kappa_{d=1}, \epsilon) = \Omega
	\crl*{\prn*{\frac{1}{\sqrt{n}}}^{\tfrac{\kappa}{\kappa - 1}} +  \prn*{\frac{1}{n \diffp}}^{\tfrac{\kappa}{\kappa - 1}}}.
	\end{equation}
\end{restatable}

\subsection{Lower bounds under approximate privacy constraints}
\label{sec:lb-appr}

We conclude our treatment by providing lower bounds but now under
\emph{approximate} privacy constraints, demonstrating the optimality of the risk
bound of Theorem~\ref{thm:sco-growth-appr}. We prove the result via a
reduction: we show that if one solves ERM with $\kappa$-growth with
error $\Delta$, this implies that one solves arbitrary convex ERM with error
$\phi(\Delta)$. Given that a lower bound of $\Omega\prn{\sqrt{d}/(n\diffp)}$
holds for ERM, a lower bound of $\phi^{-1}\prn{\sqrt{d}/(n\diffp)}$ holds for
ERM with $\kappa$-growth. However, for this reduction to hold, we require that
$\kappa \ge 2$. Furthermore, we consider $\kappa$ to be roughly a constant---in
the case that $\kappa$ is too large, standard lower bounds on general convex
functions apply.

\begin{theorem}[Private lower bound for $(\diffp, \delta)$-DP]
	\label{thm:lb-appr}
  Let $\kappa \ge 2$ such that $\kappa = \Theta(1)$, $\mc{X} =
  \ball_2^d(\rad)$. Let $d \ge 1$ and $\S = \crl{\pm 1/\sqrt{d}}^d$. Assume that
  $n\diffp = \Omega(\sqrt{d})$, then for any $(\diffp, \delta)$ mechanism
  $\msf{A}$, there exists $\lambda > 0, F$ and $\mc{S} \subset \S$ such that
  \begin{equation*}
    \E\brk{\femp(\msf{A}(\mc{S}))} - \inf_{x'\in\mc{X}}\femp(x') \ge \tilde{\Omega}\brk*{\frac{1}{\lambdae}
      \prn*{\frac{L\sqrt{d}}{n\diffp}}^{\tfrac{\kappa}{\kappa-1}}
    }.
  \end{equation*}
\end{theorem}

\Cref{thm:lb-appr} implies that the same lower bound (up to logarithmic factors) applies to SCO
via the reduction of~\cite[][Appendix C]{BassilyFeTaTh19}. Before proving the
theorem, let us state (and prove in~\Cref{app:prf-lb-approx}) the following
reduction: if an \ensuremath{(\diffp,\delta)}-DP algorithm achieves excess error (roughly) $\Delta$ on
ERM for any function with $\kappa$-growth, there exists an \ensuremath{(\diffp,\delta)}-DP algorithm that
achieves error $\Delta^{(\kappa-1) / \kappa}$ for any convex function. We
construct the latter by iteratively solving ERM problems with geometrically
increasing $\norm{\cdot}_2^\kappa$-regularization towards the previous iterate
to ensure the objective has $\kappa$-growth.

\begin{restatable}[Solving ERM with $\kappa$-growth implies solving any convex
  ERM]{proposition}{restatePropReduction}\label{prop:reduction-growth-to-erm}
  Let $\kappa \ge 2$. Assume there
  exists an $(\epsilon, \delta)$ mechanism $\msf{A}$ such that for any
  $\lip$-Lipschitz loss $G$ on $\mc{Y}$ and dataset $\mc{S}$ such that
  $g_{\mc{S}}(x) \defeq \frac{1}{n}\sum_{s\in\mc{S}} G(x;s)$ exhibits
  $(\lambda, \kappa)$-growth, the mechanism achieves excess loss
  \begin{equation*}
    \E\brk{\gemp(\msf{A}(\mc{S}, G, \mc{Y}))} - \inf_{y'\in\mc{Y}}\gemp(y')
    \le \frac{1}{\lambdae}\Delta(n, L, \epsilon, \delta).
  \end{equation*}

  Then, we can construct an $(\diffp, \delta)$-DP mechanism $\msf{A'}$ such that
  for any $\lip$-Lipschitz loss $f$, the mechanism achieves excess loss
  \begin{equation*}
    \E\brk{\femp(\msf{A}'(\mc{S}))} - \inf_{x'\in\mc{X}}\femp(x')
    \le O\prn*{D\brk*{\Delta(n, L, \epsilon / k, \delta/k)}^{\tfrac{\kappa - 1}{\kappa}}},
  \end{equation*}
  where $k$ is the smallest integer such that
  $k \ge \log\brk*{\frac{\kappa^{\tfrac{1}{\kappa -
          1}}L^{\tfrac{\kappa}{\kappa-1}}}{2^{2\kappa - 3}\Delta(n, L, \diffp/k,
      \delta/k)}}$.
\end{restatable}

With this proposition, the proof of the theorem directly follows
as~\citet{BassilySmTh14} prove a lower bound $\Omega(\sqrt{d}/(n\diffp))$ for
ERM with \ensuremath{(\diffp,\delta)}-DP.

\section*{Discussion}

In this work, we develop private algorithms that adapt to the growth of the
function at hand, achieving the convergence rate corresponding to the
``easiest'' sub-class the function belongs to. However, the picture is not yet
complete. First of, there are still gaps in our theoretical understanding, the
most interesting one being $\kappa = 1$. On these functions, appropriate
optimization algorithms achieve linear convergence~\cite{XuLiYa17} and raise the
question, can we achieve exponentially small privacy cost in our setting?
Finally, while our optimality guarantees are more fine-grained than the usual
minimax results over convex functions, they are still contigent on some
predetermined choice of sub-classes. Studying more general notions of adaptivity
is an important future direction in private optimization.

\arxiv{

  \subsection*{Acknowledgments}
  The authors would like to thank Karan Chadha and Gary Cheng for comments on an
  early version of the draft.  }

redpajama_set_name: RedPajamaArXiv