When it comes to the Greek language, we actually know more words than we think we do. In the book titled "You Speak Greek, You Just Don't Know It" by Annie Stefanides, there are more than 6,000 Greek words listed that are used in the English vocabulary, but the author claims that these are merely examples of the 40,000 Greek words used in the English language. Every word we speak has a meaning of which the essence may be rooted much deeper than you think.
The Greek language is a branch of the Indo-European language family and English evolved from this language family. It is often difficult to pinpoint the exact roots of a language due to the mix of influences but it is possible through comparative studies to find similarities in different Indo-European branches. English words of Greek origins are usually a hybrid between Latin and Greek which makes it hard to determine exactly how many words are directly derived from Greek, but about 80% of all English words are from Greek and Latin origin with an estimated 40,000+ words of Greek origin. Think of words such as "rhetorical" from the Greek "ritorikos" and "skeptical" from the Greek "skeptikos". The Greek influence can be seen in the "rhetoric? and "skeptic" but the "-al" is derived from Latin.
Greek is also known as the prime source of European academic language. This is especially true in Medicine where Greek is referred to as the language of Medicine with up to 90% of words derived from Greek. It doesn't stop there as Greek can also be found in many other disciplines including politics, literature, music, science, theater, and zoology to mention a few. Think of words such as "dialogue" derived from "dialogos", "lyric" from "lirikos", "metaphor" from "metaphora", and "anaesthesia" from "anesthisia". Even Greek prefixes, suffixes and stems are found in English, for example, "syn-" from the Greek "sin" meaning plus, with or together which can be seen in the words such as "synchronise" or "synergy".
Zolotas continued the first speech with: "Kyrie, I eulogise the archons of the Panethnic Numismatic Thesaurus and the Ecumenical Trapeza for the orthodoxy of their axioms, methods and policies, although there is an episode of cacophony of the Trapeza with Hellas. With enthusiasm, we dialogue and synagonise at the synods of our didymous organisations in which polymorphous economic ideas and dogmas are analysed and synthesised. Our critical problems such as the numismatic plethora generate some agony and melancholy. This phenomenon is characteristic of our epoch. But, to my thesis, we have the dynamism to program therapeutic practices as a prophylaxis from chaos and catastrophe. In parallel, a Panethnic unhypocritical economic synergy and harmonisation in a democratic climate is basic. I apologise for my eccentric monologue. I emphasise my euharistia to you, Kyrie to the eugenic and generous American Ethnos and to the organisers and protagonists of his Amphictyony and the gastronomic symposia.
The second speech: "Kyrie, it is Zeus' anathema on our epoch for the dynamism of our economies and the heresy of our economic methods and policies that we should agonise the Scylla of numismatic plethora and the Charybdis of economic anaemia. It is not my idiosyncrasy to be ironic or sarcastic, but my diagnosis would be that politicians are rather cryptoplethorists. Although they emphatically stigmatise numismatic plethora, they energise it through their tactics and practices.
Our policies have to be based more on economic and less on political criteria. Our gnomon has to be a metron between political, strategic and philanthropic scopes. Political magic has always been anti-economic. In an epoch characterised by monopolies, oligopolies, monopsonies, monopolistic antagonism and polymorphous inelasticities, our policies have to be more orthological. But this should not be metamorphosed into plethorophobia, which is endemic among academic economists. Numismatic symmetry should not hyper-antagonise economic acme. A greater harmonisation between the practices of the economic and numismatic archons is basic.
Parallel to this, we have to synchronise and harmonise more and more our economic and numismatic policies panethnically. These scopes are more practicable now when the prognostics of the political and economic barometer are halcyonic. The history of our didymous organisations in this sphere has been didactic and their gnostic practices will always be a tonic to the polyonymous and idiomorphous ethnical economies. The genesis of the programmed organisation will dynamise these policies. Therefore, I sympathise, although not without criticism on one or two themes, with the apostles and the hierarchy of our organs in their zeal to program orthodox economic and numismatic policies, although I have some logomachy with them.
So, next time you hear the saying "it's all Greek to me" referring to something being incomprehensible or not understandable, just remember that in the literal sense of the phrase it may truly be all Greek!
Skylla or Scylla: together with Charybdis, monsters of Greek mythology, mentioned by Homer. They lived in the strait of Messina (Sicily). To be between Skylla and Charybdis = to deal with two perilous alternatives.
Give us your email and we'll keep you in the loop.


2) How does it go about that?
3) What does your game reward/discourage/encourage?
Your game needs to be about something. It can be leveling up a character to face greater challenges, it could be about the stress of working in a business environment, it could be about protecting people unable/unwilling to protect themselves. But it needs to be about something. I call this the premise.
The game (mechanics, setting, everything) should all work toward establishing and reinforcing that premise. If your game is about leveling up, make that MEAN SOMETHING. If the game is about tragic romance, have rules for flirtation and for breaking up with people. Elements are all the pieces of your game.
Rewards drive play. This is the "engine" of the game. D&D uses XP's as a base (because XP's are used to level the character) but it also uses money, equipment and context (better tactics, new allies, enemies, knowledge, etc.). The System is how the Elements fit together to establish and reinforce the Premise. This is the core of Game Design.
Answer these and your game will work. Fail to answer them and it won't.
Thanks for posting this. I've been asking myself some very similar questions while working on refining Spookybeans with Dregg, but this is nicely focused.
I would want to be careful about the "reward" aspect of it, IMHO. There are intrinsic and extrinsic motivations, and "reward" usually means extrinsic motivation. Do this and you'll get that. Extrinsic motivation for an activity tends to decrease people's motivation for and performance in that activity.
I'd prefer to think not in terms of "what does it reward?" but "what does it enable?" And your list of "rewards" moves in that direction -- everything mentioned is a means to open up one's options and enable new possibilities in play.


Beyond the realms of decorating.
Next thing I know, I am outside helping her friend, who is over from Portland in America (a really, really nice guy) helping him to put together a Hozelock hose system, and then help in unblocking the drain outside and giving it a good blasting with the new hose.
I am not a man you want for that sort of thing as I am a bit weak stomached when it comes to poo smells!
We had a good giggle while doing it, did me and Jim. He got flicked with "stuff" here and there and, I laughed.
That is another item to add to my book, Life as a Decorator!
« Protecting your furniture and flooring when decorating.


Since his first EP "Cold Wind" was released in November, Joshua Hyslop has focused the messages of his music around hope and positivity, and his newest album "In Deepest Blue" is no exception to that trend.
Hyslop is quoted saying the deepest blue is meant to be "the calmness we all have" and "the hope inside," creating an album that sounds like Ed Sheeran collaborated with Iron & Wine and centered their album's message on optimism.
Born in British Columbia and raised in Canada, Hyslop is incredibly optimistic and possesses great interpersonal abilities including traveling across the country, spending quality time with families who invite him in and sharing his message everywhere he goes.
"In Deepest Blue" is a pretty straightforward album. Hyslop is a Christian artist, and you can see the religious influence in his works.
The album opens with a track titled "The Flood," possessing a voice soft enough to coincide with the softness of his music. The lyrics are based off of the biblical story "The Return of the Prodigal Son," setting a general theme for the songs coming after. Later in the album is the track titled "Living and Dying," which gives off an incredibly strong Iron & Wine vibe with lyrics describing Hylsop's underlying fear of human mortality.
The next track "Let It Go" (not a reference to "Frozen") and the second to last track titled "Gone" are a response to "Living and Dying." Within these tracks, Hyslop is talking about relinquishing those worries and just enjoying life because "the lord knows/ [he'll] be gone" eventually, and it is inevitable.
Because of his straightforwardness, he named the most important song of the album after the album itself. "In Deepest Blue" seems to be directed to an individual, and the remainder of the album also seems directed toward that same person.
Hylsop shows off his fingerpicking and guitar skills in the instrumental track titled "Instrumental" (not much thought put into that title, but it's still a solid track).
The album finally ends with a track titled "Tonight," which ties everything together by tying in references to previous messages heard in the album: accepting help when it is offered and finding the light in the darkness.
As a Christian, indie, folk artist, his work is the center of the genre and is unique when compared to other Christian artists. As an indie, folk artist alone, he is recommendable for those who enjoy a style similar to Iron & Wine and Jack Johnson.


This reading practice simulates one part of the IELTS Academic Reading test. You should spend about twenty minutes on it. Read the passage and answer questions 13-25.
13. The early public perception of Antarctica was of a place with little underlying worth.
14. Antarctica was once very different from what it is today.
15. The extreme environment of Antarctica influences weather patterns throughout the southern hemisphere.
16. Companies in Australia use research carried out in the Antarctic to plan their business.
17. When sea ice in the Antarctic is less widespread, there is more food available for other marine animals.
18. Whales and seabird spend an equal amount of time feeding in both the northern and southern hemispheres.
19. The most ferocious storms ever recorded have occurred in Antarctica.
20. Sea ice tends to remain static for long periods of time.
21. The positive effects of cold water from Antarctica is crucial to other oceans.
Write your answers in 22-25 on your answer sheet.
Arctic krill is part of the of many other marine creatures.
Extreme winds cause ice to be blown off the surface of the sea leaving large .
The condition of the northern oceans owe their to what is going on in Antarctica.
Antarctica - in from the cold?
A little over a century ago, an expedition led by Scott, Shackleton and Mawson bravely battled against Antarctica's blizzards, cold and deprivation. In the name of Empire and in an age of heroic deeds they created an image of Antarctica that was to last well into the 20th century—an image of remoteness, hardship, bleakness and isolation that was the province of only the most courageous of men. The image was one of a place removed from everyday reality, of a place with no apparent value to anyone.
As we enter the 21st century, our perception of Antarctica has changed. Although physically Antarctica is no closer and probably no warmer, and to spend time there still demands a dedication not seen in ordinary life, the continent and its surrounding ocean are increasingly seen to be an integral part of planet earth, and a key component in the earth's ecosystem. Is this because the world seems a little smaller these days, shrunk by TV and tourism, or is it because Antarctica really does occupy a central spot on Earth's mantle? Scientific research during the past half century has revealed—and continues to reveal—that Antarctica's great mass and low temperature exert a major influence on climate and ocean circulation, factors which influence the lives of millions of people all over the globe.
Antarctica was not always cold. The slow break-up of the super-continent Gondwana with the northward movements of Africa, South America, India and Australia eventually created enough space around Antarctica for the development of an Antarctic Circumpolar Current (ACC), that flowed from west to east under the influence of the prevailing westerly winds. Antarctica cooled, its vegetation perished, glaciation began, and the continent took on its present-day appearance. Today the ice that overlies the bedrock is up to 4km thick, and surface temperatures as low as -89.2deg C have been recorded. The icy blast that howls over the ice cap and out to sea—the so-called katabatic wind—can reach 300 km/hr, creating fearsome wind-chill effects.
Out of this extreme environment come some powerful forces that reverberate around the world. The Earth's rotation, coupled to the generation of cells of low pressure off the Antarctic coast, allow astronauts in space a view of Antarctica that is as beautiful as it is awesome. Spinning away to the northeast, the cells grow and deepen, whipping up the Southern Ocean into the mountainous seas so respected by mariners. Recent work shows that the temperature of the ocean may be a better predictor of rainfall in Australia than is the pressure difference between Darwin and Tahiti, the Southern Oscillation Index. By receiving more accurate predictions, graziers in northern Queensland are able to avoid overstocking in years when rainfall will be poor. Not only does this limit their losses but it prevents serious pasture degradation that may take decades to repair. CSIRO is developing this as a prototype forecasting system, but we can confidently predict that as we know more about the Antarctic and Southern Ocean we will be able to enhance and extend our predictive ability.
The ocean's surface temperature results from the interplay between deep-water temperature, air temperature and ice. Each winter between 4 and 19 million square km of sea ice form, locking up huge quantities of heat close to the continent. Only now can we start to unravel the influence of sea ice on the weather that is experienced in southern Australia. But in another way the extent of sea ice extends its influence far beyond Antarctica. Antarctic krill—the small shrimp-like crustaceans that are the staple diet for baleen whales, penguins, some seals, flighted sea birds and many fish—breed well in years when sea ice is extensive and poorly when it is not. Many species of baleen whales and flighted sea birds migrate between the hemispheres and when the krill are less abundant they are less able to thrive.
The circulatory system of the world's oceans is like a huge conveyor belt, moving water and dissolved minerals and nutrients from one hemisphere to the other, and from the ocean's abyssal depths to the surface. The Antarctic Circumpolar Current (ACC) is the longest current in the world and has the largest flow. Through it, the deep flows of the Atlantic, Indian and Pacific Oceans are joined to form part of a single global thermohaline circulation. During Winter, the howling katabatics sometimes scour the ice off patches of the sea's surface leaving large ice-locked lagoons, or 'polynyas'. Recent research has shown that as fresh sea ice forms, it is continuously stripped away by the wind and may be blown up to 90km in a single day. Since only fresh water freezes into ice, the water that remains becomes increasingly salty and dense, sinking until it spills over the continental shelf. Cold water carries more oxygen than warm water, so when it rises, well into the northern hemisphere, it re-oxygenates and revitalizes the ocean. The state of the northern oceans, and their biological productivity, owe much to what happens in the Antarctic.


Adv. Assaf Harel, partner, who leads our firm's Cyber-Security and Privacy practice, addressed, in an interview to Bloomberg BNA, Facebook's preparations for Israel's general elections.
Read Gornitzky & Co.'s latest publications.


text:
« Ask a Wench -- What's it like to create an interwoven anthology? | Main | A Historic Dinner »
Serial stories—a what a novel idea!
Cara/Andrea here, We've recently been talking here about our writing processes, and how time, ambiance, computer vs pen and paper affect how we create our stories. And deadlines—those were mentioned a lot too! In thinking about the subject, it seems to me that some things have been universal angts to writers across the ages. Inspiration. Struggle. How the words flow. And yes, the dreaded deadline! It's never easy. But on reading a recent article on Charles Dickens and his The Mystery of Edwin Drood, it reminded me that during the 19th century, a phenomena developed—one that proved hugely popular with the public—that put even more pressure on authors.
Dickens, who was forced to leave home and find work in a factory after his father was put in debtor's prison, helped pioneer the "serial" novel, which involved publishing the book in either weekly or monthly installments within the pages of a high-circulation popular magazine of the times. Pickwick Papers, his first book to be serialized, was showcased to the public in 1836, and launched the author on his way to literary fame.
Serialization was an interesting confluence of tech and creativity. The Industrial Revolution had spread to the publishing world, and the new high-powered steam presses were making possible the mass production of inexpensive magazines and newspapers. It's said that Dickens would gauge reader reaction to the story and in response would actually noodle with his original plot to meet. For example, word is he altered Miss Mowcher in David Copperfield, to be shown in a better light.
Now, back to "Drood." His works were so popular that The Mystery of Edwin Drood created a huge crisis for his readers. Designed to be released in 12 parts, the book, which had a very complicated plot revolving around opium dens, a romance complete with evil characters, and the sudden disappearance of the hero—had just released part 6 when Dickens inconveniently passed away without having completed the story! There have been many attempts by others to finish the book since then—including one in 1873 by a printer in Vermont who claimed to have the words dictated to him by the spirit of Dickens. Um, talk about giving new meaning to the term "ghostwriter."
As it is to this day, publishers were on the lookout for "hooks" to attract readers, and in the 19th century, the serialized novel proved a huge draw. People who might not be able to buy books could afford the small cost of the magazine, and thus over time read the whole book. In many ways, it brought a whole new audience to the literary world, and by the late Victorian era, many of the top authors, including Wilkie Collins and Sir Arthur Conan Doyle were releasing their works in this form.
England was not the only country where serial novels proved wildly popular. In France, Alexandre Dumas enthusiastically embraced the new form. The Three Musketeers and The Count of Monte Cristo (which needed 139 installments) were published as feuilletons, and at times he was working 14 hours a day on several serializations at once. (Okay, shoot me now!) Flaubert followed suit, releasing Madame Bovary as a serial in 1856. In Russia, Tolstoy published Anna Karenina and The Brothers Karamazov within the pages of the Russian Messenger.
Serialization proved to be very popular in America as well. In 1851, Uncle Tom's Cabin, by Harriet Beecher Stowe was published in parts by the Abolitionist magazine, The National Era, and its widespread availability helped influence public opinion on slavery. Harper's and Atlantic Monthly developed a reputation as strong literary forums by publishing the likes of Henry James and Herman Melville.
With the advent of radio and television, the serialized novel soon lost its allure—though it's interesting to note that in 1984, Tom Wolfe's The Bonfire of the Vanities was published as a 29-part serial in Rolling Stone magazine, partly, it's said, as an homage to Dickens.
Today, given ever-shrinking attention spans, I'm not sure serials would find a willing audience. But what do you think? Would you enjoy following a serial novel? Or would you be too impatient to wait for the release of each segment. I tend to like to race through a book I love, so not sure how I would like it. However, the notion is intriguing . . . hmmm, maybe the Wenches should start a serial novel on the blog! What do you think? (ducking flying rotten tomatoes from the other Wenches! <G>)
Posted by Andrea Penrose on Thursday, September 17, 2015 at 10:00 PM in Andrea Penrose, Andrea Pickens, Cara Elliott, History, Research, Writing Topics | Permalink

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Filter and sort inputed list of integers for non negative numbers python I am trying to take a list that a user inputs, filter and sort in ascending order but only for the non-negative numbers.  I thought I understood how to do this but it will not work with my logic.
my_list = []
n = int(input())
for i in range(0, n):
    element = int(input())
    if element > 0:
        my_list.append(element)
my_list.sort()
print(my_list)

Here is the error I receive:
ValueError: invalid literal for int() with base 10: '10 -7 4 39 -6 12 2' It doesn't like line 2

A: user_input = input()

my_list = [int(i) for i in user_input.split() if (int(i)>=0)]

my_list.sort()

[print(i, end=' ') for i in my_list]

A: # this solution works
user_input = input()
my_list = [int(i) for i in user_input.split() if (int(i)>=0)]
my_list.sort()
[print(i, end=' ') for i in my_list]

A: input() returns the entire string at once, not the individual elements, so you need to split your input into the individual numbers using str.split, then convert them to integers (this can be done conveniently using map) before selecting only the non-negative numbers to put into my_list (this can be written as a list comprehension). You can then sort the list with list.sort:
elements = map(int, input().split())
my_list = [e for e in elements if e >= 0]
my_list.sort()
print(my_list)

For input of 10 -7 4 39 -6 12 2, this produces
[2, 4, 10, 12, 39]

Demo on ideone

A: I'm a newbie, but there is probably a better way to write this.
interger = input().split()

new_interger = []
sorted_list = []

for x in interger:
    new_interger.append(int(x))
    for x in new_interger:
        if x >= 0:
            new_interger.pop()
            sorted_list.append(x)
sorted_list.sort()

for x in sorted_list:
    print(x, end=' ')

A: I broke it up a little to display the steps required to output the elements of the list rather than the actual list:
data = input()

#turn input to a list based on spaces
dat_list = data.split()
#turn elements in list to integers
dat_list = [int(x) for x in dat_list]
#fix list to only include positive integers
dat_list = [i for i in dat_list if pn >= 0]

#sort list
dat_list.sort()

#print each integer in list with a space after
for i in dat_list:
    print(i, end=' ')

Tried to be as explicative as possible for people having a hard time understanding, meaning this is not the shortest or most effective way of doing this, but newbie friendly

A: You have to put the lenght of the array into n and then use the i do define the element:
my_list = []
n = len(input().split())
for i in range(0, n):
    element = int(input().split()[i])
    if element > 0:
        my_list.append(element)
my_list.sort()
print(my_list)

print:
[2, 4, 10, 12, 39]

EDIT: The upper version was tested and works with Py 3.8.2
This is working also on https://ideone.com/rDw1w6#stdin using Py 3.7.3
my_list = []
my_string_array = input().split()
n = len(my_string_array)
for i in range(0, n):
    element = int(my_string_array[i])
    if element > 0:
        my_list.append(element)
my_list.sort()
print(my_list)

redpajama_set_name: RedPajamaStackExchange


Whales beached and some died in the Bahamas in 2000 when Navy vessels used mid-frequency sonar in the area. The council said there had been many mass strandings of whales related to sonar and that many of the whales suffered injuries including bleeding around the brain and ears.
The waters off Southern California are home to five endangered whale species.
The Supreme Court noted that Southern California is important for the Navy's training because it's the only area on the West Coast that's relatively close to land, air and sea bases and amphibious landing areas.
"The Navy's need to conduct realistic training with active sonar to respond to the threat posed by enemy submarines plainly outweighs" the arguments advanced by the environmentalists, the Supreme Court said in an opinion written by Chief Justice John G. Roberts. The opinion called the sonar mission crucial because it's the only proven method of identifying submerged diesel-electric submarines operating on battery power.
"For the plaintiffs, the most serious possible injury would be harm to an unknown number of the marine mammals that they study and observe. In contrast, forcing the Navy to deploy an inadequately trained antisubmarine force jeopardizes the safety of the fleet," Roberts wrote.
The decision wasn't entirely clear-cut. Justices Stephen Breyer and John Paul Stevens, writing separately, agreed that the lower courts hadn't sufficiently explained their reasoning, and Breyer said he would've imposed tougher restrictions. Justices Ruth Bader Ginsburg and David Souter dissented altogether.
The Navy has taken steps since January 2007 to protect marine mammals during sonar exercises. However, it objected to two measures imposed by a lower court: stopping sonar use when marine mammals were spotted within 2,200 yards and powering down the sonar under certain other conditions.
The Bush administration's Council on Environmental Quality declared in January that "emergency circumstances" existed that should dissolve those limitations. In March, the San Francisco-based 9th U.S. Circuit Court of Appeals upheld the lower court's requirement that the Navy take specific steps to protect whales. The Supreme Court ruling reverses that decision.
The Natural Resources Defense Council said Wednesday's ruling wasn't a total loss for its defense of the whales.
Four of six measures to protect marine mammals that the lower court put in place remain in effect. They include a 12-nautical-mile buffer zone off the coast, protecting shallower waters where marine mammals congregate, said Zak Smith, an attorney for the council who worked on the case.
In addition, the Navy isn't allowed to train in an area of the ocean between Santa Catalina and San Clemente islands that's important for marine mammals.
"We're not opposed to the Navy's use of sonar. It's just there are better places to use it instead of areas of rich biological importance," Smith said.


Embo House is starting to show its capability as a Dornoch wedding venue, with the very lovely Agne and Jerry using it as the base for their celebrations in October 2018. Luckily, it was a fabulously dry and warm weekend, even though it was nearly November.
I was thrilled to be able to be around to take a few snaps, as preparations got underway. A marquee was erected in the back garden, on the circular lawn and its transparent sides allowed the sunshine and the garden views in.
It is also possible to erect a tent on the front lawn, looking out to the Dornoch Firth, but this couple decided that was where they wanted to take photos, as their guests (and priests) looked on! These two had previously been married in St. Finnbarr's Scottish Episcopal Church in Dornoch, though there are in fact a number of churches in the town, (plus of course the beautiful, 13th century Cathedral) to choose from. And don't forget that in Scotland you can also be married anywhere you can get the local registrar to agree with, including in Embo House, or on the beach.
…and as Jerry decided to carry his bride towards the sea, for some more "rural" shots!
Meanwhile, back at the house, the food was nearly ready, drinks were flowing and the party was about to begin.
As I left our guests to their party, the evening sun came out with real warmth and I reflected what a lovely Dornoch wedding venue this really is.
If you would like to use Embo House as a Dornoch wedding venue, or indeed as a venue for other kinds of events, please do get in touch. We would be delighted to discuss your requirements with you and see what we can do to accommodate you.
https://embohouse.co.uk/wp-content/uploads/2018/12/1-sm-1.jpg 3197 4819 Ginny Knox https://embohouse.co.uk/wp-content/uploads/2018/01/logo-white-bigger.jpg Ginny Knox2018-12-06 09:24:082018-12-10 11:33:16Embo House is now a Dornoch wedding venue!


Leaving the company you founded is always a tough decision and a tough process that involves many people. It requires a series of potentially emotional "crucial-conversations." It is actually not that uncommon in venture-backed companies for one or more of the original founders to leave at some point. There is a decent article on the topic here: https://hbswk.hbs.edu/item/the-founding-ceos-dilemma-stay-or-go.
Still it is extremely difficult to let go. You live and breathe the company you start. Years of working to connect as many people as possible to the dream gives you a feeling of "ownership" and connection that no stock certificate can replace. Starting a company is a lot of work. It takes a lot of effort. There are many decisions to make and many voices to incorporate. Hiring, firing, raising money, engaging customers, engaging employees, planning projects, organizing events, and aligning a pastiche of personalities while staying relevant in a rapidly evolving technology jungle is difficult.
As a founder over 40 with modest means, I had a family of 6 children who relied on me. That family had teenage children who needed my attention and pre-school and elementary-school children that I could not simply leave only in the hands of my wife. I look back and sometimes wonder how we pulled it off. The truth probably lies in the time we borrowed: time from exercise, time from sleep, time from vacations, and time from family. I'd like to say that this dissonance against "work-life-harmony" was always a bad choice, but honestly, I don't see how I could have made too may different choices and still have created Anaconda.
Several things drove me. I could not let the people associated with the company down. I would not lose the money for those that invested in us. I could not let down the people who worked their tail off to build manage, document, market, and sell the technology and products that we produced. Furthermore, I would not let the community of customers and users down that had enabled us to continue to thrive.
The only way you succeed as a founder is through your customers being served by the efforts of those who surround you. It is only the efforts of the talented people who joined us in our journey that has allowed Anaconda to succeed so far. It is critical to stay focused on what is in the best interests of those people.
Back in 2012, Peter and I knew a few things: 1) we had to connect Python to the Big Data movement; 2) we needed to help the scientific programmer, or a data-scientist developer build visualization-based applications quickly in the web; and 3) we needed to scale the stack of code around the PyData community to bigger hardware and multiple machines. We had big visions of an interconnected data-web, distributed schedulers, and data-structures that traversed the internet which could be analyzed across the cloud with simple Python scripts. We talked and talked about these things and grew misty-eyed in our enthusiasm for the potential of what was possible if we just built the right technology and sold just the right product to fund it.
We knew that we wanted to build a product-company -- though we didn't know exactly what those products would be at the outset. We had some ideas, only portions of which actually worked out. I knew how to run a consulting and training company around Python and open-source. Because of this, I felt comfortable raising money from family members. While consulting companies are not "high-growth" they can make real returns for investors. I was pretty confident that I would not lose their money.
We raised $2.25million from a few dozen investors consisting of Peter's family, my family, and a host of third-parties from our mutual networks. Peter's family was critical to this early stage because they basically "led the early round" and ensured that we could get off the ground. After they put their money in the bank, we could finish raising the rest of the seed round which took about 6 months to finish.
It is interesting (and somewhat embarrassing and so not detailed here) to go back and look at what products we thought we would be making. Some of the technologies we ended up building (like Excel integration, Numba, Bokeh, and Dask) were reflected in those early product dreams. However, the real products and commercial success that Anaconda has had so far are only a vague resemblance to what we thought we would do.
Building a Python distribution was the last thing on our minds. I had been building Python distributions since I released SciPy in 2001. As I have often repeated, SciPy was actually the first Python distribution masquerading as a library. The single biggest effort in releasing SciPy was building the binary installers and making sure everything compiled well. With Fortran compilers still more scarce than they should be, it can still be difficult to compile and build SciPy.
Fortunately, with conda, conda-forge, and Anaconda, along with the emergence of wheels, almost nobody needs to build SciPy anymore. It is so easy today to get started with a data-science project and get all the software you need to do amazing work fast. You still have to work to maintain your dependencies and keep that workflow reproducible. But, I'm so happy that Anaconda makes that relatively straightforward today.
This was only possible because General Catalyst and BuildGroup joined us in the journey in the spring of 2015 to really grow the Anaconda story. Their investment allowed us to 1) convert to a serious product-company from a bootstrapped consulting company with a few small products and 2) continue to invest heavily in conda, conda-forge, and Anaconda.
There is nothing like real-world experience as a teacher, and the challenge of converting to a serious product company was a tremendous experience that taught me a great deal. I'm grateful to all the people who brought their best to the company and taught me everyday. It was a privilege and an honor to be a part of their success. I am grateful for their patience with me as my "learning experiences" often led to real struggles for them.
There are many lasting learnings that I look forward to applying in future endeavors. The one that deserves mention in this post, however, is that building enterprise software that helps open-source communities should be done by selling a complementary product to the open-source. The "open-core" model does not work as well. I'm a firm believer that there will always be software to sell, but infrastructure should be and will be open-source --- sustained vibrantly from the companies that depend on it. Joel Spolsky has written about complementary products before. You should read his exposition.
Early on at Anaconda, Peter and I decided to be a board-led company. This board which includes Peter and I has the final say in company leadership and made the important decision to transition Anaconda from being founder-led to being led by a more experienced CEO. After this transition and through multiple conversations over many months we all concluded that the best course of action that would maximize my energy and passion while also allowing Anaconda to focus on its next chapter would be for me to spin-out of Anaconda and start a new services and open-source company where I could pursue a broader mission.
This new company is Quansight (short for Quantitative Insight). Our place-holder homepage is at http://www.quansight.com and we are @quansightai on Twitter. I'm excited to tell you more about the company in future blog-posts and announcements. A few paragraphs will suffice for now.
Our overall mission is to develop people, build technology, and discover products to empower people with knowledge and data to solve the world's most challenging problems. We are doing that currently by connecting organizations sustainably with open source communities to solve their hardest problems by enabling teams to transparently apply science to their data.
One of the things we are doing is to help companies get started with AI and ML by applying the entire PyData stack to the fundamental data organization, data visualization, and model management problem that is required for practical success with ML and AI in business. We also help companies generally improve their data-science practice by leveraging all the power of the Python, PyData, and related ecoystems.
We are also hard at work on the sustainability problem by continuing the tradition we started at Continuum Analytics of building successful and sustainable open-source "practices" that synchronize company needs with open-source technology development. We have some innovative business approaches to this that we will be announcing in the coming weeks and months.
I'm excited that we have several devs working hard to help bring JupyterLab to 1.0 this year along with a vibrant community. There are many exciting extensions to this remarkable platform that remain to be written.
We also expect to continue to contribute to the PyViz activities that continue to explode in the Python ecosystem as visualization is a critical first step to understanding and using any data you care about.
Finally, Stefan Krah has joined us at Quansight. Stefan is an award-winning Python core developer who has been steadily working over the past 18 months on a small but powerful collection of projects collectively called Plures. These will be more broadly available in the next few months and published under the xnd brand. Xnd is a generic container concept in C with a Python binding that together with its siblings ndtypes and gumath allows building flexible array-computing pipelines over many kinds of data-types.
This technology will serve to underly any array-computing framework and be a glue between machine-learning and data-science frameworks of all kinds. Our plan is to use this tool to help reduce the data and computational silos that currently exist across the open-source ecosystem.
There is still much to work on and many more technologies to emerge. It's an exciting time to work in machine learning, data-science, and scientific computing. I'm thrilled that I continue to get the opportunity to be part of it. Let me know if you'd like to be a part of our journey.
NumFOCUS just finished its 5th year of operations, and I've lately been reflective on the early days and some of the struggles we went through to get the organization started. It once was just an idea in a few community-minded developer's heads and now exists as an important non-profit Foundation for Open Data Science, democratic and reproducible discovery, and a champion for technical progress through diversity.
When Peter Wang and I started Continuum in early 2012, I had already started the ball rolling to create NumFOCUS. I knew that we needed to create a non-profit that would create leadership and be a focus of community activity outside of any one company. I strongly believe that for open-source to thrive, full-time attention needs to be paid to it by many people. This requires money. With the tremendous interest in and explosion around the NumPy community, it was clear to me that this federation of loosely-coupled people needed some kind of organization that could be community-led and could be a rallying point for community activity and community-led financing. The potential also exists for NumFOCUS to act as community-based accountability to encourage positively re-inforcing behavior in the open-source communities it intersects with.
In late 2011, I started a new mailing list and invited anyone interested in discussing the idea of an independent community-run organization to the list. Over 100 people responded and so I knew there was interest. We debated on that list what to call the new concept for several weeks and Anthony Scopatz's name "NumFOCUS" stuck as the best alternative over several other names. As an acronym, NumFOCUS could mean Numerical Foundation for Open Code and Usable Science. I created a new mailing list, and then set about creating the legal organization called NumFOCUS and filing necessary paperwork.
In December of 2011, I coordinated with Fernando Perez, Perry Greenfield, John Hunter, and Jarrod Millman who had all expressed some interest in the idea and we incorporated in Texas (using LegalZoom) and became the first board of NumFOCUS. We had a very simple set of bylaws and purposes all centered around making Science more accessible. We decided to meet every-other week. We all knew we were creating something that would last a long time.
In early 2012, I wanted to ensure NumFOCUS success and knew that it needed a strong, full-time, Executive Director to make that happen. The problem was NumFOCUS didn't have a lot of money. A few of the board members had made donations, but Continuum with its own limited means was funding the majority of the costs for getting NumFOCUS started. With the legal organization started, I created bank-accounts and setup the ability for people to donate to NumFOCUS with help from Anthony Scoptatz who was the first treasurer of NumFOCUS.
I had met Leah Silen through other community interactions in Austin back in 2007. I knew her to be a very capable and committed person and thought she might be available. I asked her if she would come aboard and be employed by Continuum but work full-time for NumFOCUS and the new board. She accepted and the organization of NumFOCUS began to improve immediately.
With her help, we transitioned the organization from LegalZoom's beginnings to register directly with the secretary of state in Texas and started the application process to become a 501(c)3. She also quickly became involved in organizing the PyData conferences which Continuum initially spear-headed along with Julie Steele and Edd Wilder-James (at the time from O'Reilly). In 2012, we had our first successful PyData conference at the GooglePlex in Mountain View . It was clear that PyData could be used as a mechanism to provide revenue for NumFOCUS (at least to support Leah and other administrative help).
We began working under that model through 2013 and 2014 with Continuum initially spending a lot of human resources and money organizing and running PyData with any proceeds going directly to NumFOCUS. There were no proceeds in those years except enough to help pay for Leah's salary. The rest of Leah's salary and PyData expenses came from Continuum which itself was still a small startup.
During these years of PyData growth in communities around the world, James Powell, became a drumbeat of consistency and community engagement. He has paid his own way to nearly every PyData event throughout the world. He has acted as emcee, volunteer extraordinaire, and popular speaker with his clever implementations and explanations of the Python stack.
Andy Terrel had been a friend of NumFOCUS and a member of the community and active with the board from its beginning. In 2014, while working at Continuum, he took over my board seat. In that capacity, he worked hard to gain financial independence for NumFOCUS. He was instrumental in moving PyData fully to NumFOCUS management. I was comfortable stepping back from the board and stepping down in my involvement around organizing and backing PyData from a financial perspective because I trusted Andy's leadership and non-profit management instincts. He, James Powell, Leah, and all the other local PyData meetups and organizations world-wide have done an impressive thing in self-organizing and growing the community. We should all be grateful for their efforts.
I am very proud of the work I did to help start NumFOCUS and PyData. I hope to remember it as one of the most useful things I've done professionally. I am very grateful for all the others who also helped to create NumFOCUS as well as PyData. So many have worked hard to ensure it can be a worldwide and community-governed organization to support Open Data Science for a long time to come. I'm proud of the funding and people-time that Continuum provided to get NumFOCUS and PyData started as well as the on-going support of NumFOCUS that Continuum and other industry partners continue to provide.
Now, as an adviser to the organization, I get to hear from time to time how things are going. I'm very impressed at the progress being made by the dedication of the current leadership behind Andy Terrel as President and Leah Silen as Executive Director and the rest of the current board.
If you use or appreciate any of the tools in the Open Data Science that NumFOCUS sponsors, I encourage you to join and/or make a supporting donation here: http://www.numfocus.org/support-numfocus.html. Help NumFOCUS continue its mission to support the tools and communities you rely on everyday.
Anaconda and Hadoop --- a story of the journey and where we are now.
My first real experience with cluster computing came in 1999 during my graduate school days at the Mayo Clinic. These were wonderful times. My advisor was Dr. James Greenleaf. He was very patient with allowing me to pester a bunch of IT professionals throughout the hospital to collect their aging Mac Performa machines and build my own home-grown cluster. He also let me use a bunch of space in his ultrasound lab to host the cluster for about 6 months.
The form-factor for those Mac machines really made it easy to stack them. I ended up with 28 machines in two stacks with 14 machines in each stack (all plugged into a few power strips and a standard lab-quality outlet). With the recent release of Yellow-Dog Linux, I wiped the standard OS from all the machines and installed Linux on all those Macs to create a beautiful cluster of UNIX goodness I could really get excited about. I called my system "The Orchard" and thought it would be difficult to come up with 28 different kinds of apple varieties to name each machine after. It wasn't difficult. It turns out there are over 7,500 varieties of apples grown throughout the world.
The reason I put this cluster together was to simulate Magnetic Resonance Elastography (MRE) which is a technique to visualize motion using Magnetic Resonance Imaging (MRI). I wanted to simulate the Bloch equations with a classical model for how MRI images are produced. The goal was to create a simulation model for the MRE experiment that I could then use to both understand the data and perhaps eventually use this model to determine material properties directly from the measurements using Bayesian inversion (ambitiously bypassing the standard sequential steps of inverse FFT and local-frequency estimation).
Now I just had to get all these machines to talk to each other, and then I would be poised to do anything. I read up a bit on MPI, PVM, and anything else I could find about getting computers to talk to each other. My unfamiliarity with the field left me puzzled as I tried to learn these frameworks in addition to figuring out how to solve my immediate problem. Eventually, I just settled down with a trusted UNIX book by the late W. Richard Stevens. This book explained how the internet works. I learned enough about TCP/IP and sockets so that I could write my own C++ classes representing the model. These classes communicated directly with each other over raw sockets. While using sockets directly was perhaps not the best approach, it did work and helped me understand the internet so much better. It also makes me appreciate projects like tornado and zmq that much more.
I ended up with a system that worked reasonably well, and I could simulate MRE to some manner of fidelity with about 2-6 hours of computation. This little project didn't end up being critical to my graduation path and so it was abandoned after about 6 months. I still value what I learned about C++, how abstractions can ruin performance, how to guard against that, and how to get machines to communicate with each other.
Using Numeric, Python, and my recently-linked ODE library (early SciPy), I built a simpler version of the simulator that was actually faster on one machine than my cluster-version was in C++ on 20+ machines. I certainly could have optimized the C++ code, but I could have also optimized the Python code. The Python code took me about 4 days to write, the C++ code took me about 4 weeks. This experience has markedly influenced my thinking for many years about both pre-mature parallelization and pre-mature use of C++ and other compiled languages.
Fast forward over a decade. My computer efforts until 2012 were spent on sequential array-oriented programming, creating SciPy, writing NumPy, solving inverse problems, and watching a few parallel computing paradigms emerge while I worked on projects to provide for my family. I didn't personally get to work on parallel computing problems during that time, though I always dreamed of going back and implementing this MRE simulator using a parallel construct with NumPy and SciPy directly. When I needed to do the occassional parallel computing example during this intermediate period, I would either use IPython parallel or multi-processing.
In 2012, Peter Wang and I started Continuum, created PyData, and released Anaconda. We also worked closely with members of the community to establish NumFOCUS as an independent organization. In order to give NumFOCUS the attention it deserved, we hired the indefatigable Leah Silen and donated her time entirely to the non-profit so she could work with the community to grow PyData and the Open Data Science community and ecosystem. It has been amazing to watch the community-based, organic, and independent growth of NumFOCUS. It took effort and resources to jump-start, but now it is moving along with a diverse community driving it. It is a great organization to join and contribute effort to.
A huge reason we started Continuum was to bring the NumPy stack to parallel computing --- for both scale-up (many cores) and scale-out (many nodes). We knew that we could not do this alone and it would require creating a company and rallying a community to pull it off. We worked hard to establish PyData as a conference and concept and then transitioned the effort to the community through NumFOCUS to rally the community behind the long-term mission of enabling data-, quantitative-, and computational-scientists with open-source software. To ensure everyone in the community could get the software they needed to do data science with Python quickly and painlessly, we also created Anaconda and made it freely available.
In addition to important community work, we knew that we would need to work alone on specific, hard problems to also move things forward. As part of our goals in starting Continuum we wanted to significantly improve the status of Python in the JVM-centric Hadoop world. Conda, Bokeh, Numba, and Blaze were the four technologies we started specifically related to our goals as a company beginning in 2012. Each had a relationship to parallel computing including Hadoop.
Conda enables easy creation and replication of environments built around deep and complex software dependencies that often exist in the data-scientist workflow. This is a problem on a single node --- it's an even bigger problem when you want that environment easily updated and replicated across a cluster.
Bokeh allows visualization-centric applications backed by quantitative-science to be built easily in the browser --- by non web-developers. With the release of Bokeh 0.11 it is extremely simple to create visualization-centric-web-applications and dashboards with simple Python scripts (or also R-scripts thanks to rBokeh).
With Bokeh, Python data scientists now have the power of both d3 and Shiny, all in one package. One of the driving use-cases of Bokeh was also easy visualization of large data. Connecting the visualization pipeline with large-scale cluster processing was always a goal of the project. Now, with datashader, this goal is now also being realized to visualize billions of points in seconds and display them in the browser.
Our scale-up computing efforts centered on the open-source Numba project as well as our Accelerate product. Numba has made tremendous progress in the past couple of years, and is in production use in multiple places. Many are taking advantage of numba.vectorize to create array-oriented solutions and program the GPU with ease. The CUDA Python support in Numba makes it the easiest way to program the GPU that I'm aware of. The CUDA simulator provided in Numba makes it much simpler to debug in Python the logic of CUDA-based GPU programming. The addition of parallel-contexts to numba.vectorize mean that any many-core architecture can now be exploited in Python easily. Early HSA support is also in Numba now meaning that Numba can be used to program novel hardware devices from many vendors.
The ambitious Blaze project will require another blog-post to explain its history and progress well. I will only try to summarize the project and where it's heading. Blaze came out of a combination of deep experience with industry problems in finance, oil&gas, and other quantitative domains that would benefit from a large-scale logical array solution that was easy to use and connected with the Python ecosystem. We observed that the MapReduce engine of Hadoop was definitely not what was needed. We were also aware of Spark and RDD's but felt that they too were also not general enough (nor flexible enough) for the demands of distributed array computing we encountered in those fields.
After early work trying to extend the NumPy code itself led to struggles because of both the organic complexity of the code base and the stability needs of a mature project, the Blaze effort started with an effort to re-build the core functionality of NumPy and Pandas to fix some major warts of NumPy that had been on my mind for some time. With Continuum support, Mark Wiebe decided to continue to develop a C++ library that could then be used by Python and any-other data-science language (DyND). This necessitated defining a new data-description language (datashape) that generalizes NumPy's dtype to structures of arrays (column-oriented layout) as well as variable-length strings and categorical types. This work continues today and is making rapid progress which I will leave to others to describe in more detail. I do want to say, however, that dynd is implementing my "Pluribus" vision for the future of array-oriented computing in Python. We are factoring the core capability into 3 distinct parts: the type-system (or data-declaration system), a generalized function mechanism that can interact with any "typed" memory-view or "typed" buffer, and finally the container itself. We are nearing release of a separated type-library and are working on a separate C-API to the generalized function mechanism. This is where we are heading and it will allow maximum flexibility and re-use in the dynamic and growing world of Python and data-analysis. The DyND project is worth checking out right now (if you have desires to contribute) as it has made rapid progress in the past 6 months.
As we worked on the distributed aspects of Blaze it centered on the realization that to scale array computing to many machines you fundamentally have to move code and not data. To do this well means that how the computer actually sees and makes decisions about the data must be exposed. This information is usually part of the type system that is hidden either inside the compiler, in the specifics of the data-base schema, or implied as part of the runtime. To fundamentally solve the problem of moving code to data in a general way, a first-class and wide-spread data-description language must be created and made available. Python users will recognize that a subset of this kind of information is contained in the struct module (the struct "format" strings), in the Python 3 extended buffer protocol definition (PEP 3118), and in NumPy's dtype system. Extending these concepts to any language is the purpose of datashape.
In addition, run-times that understand this information and can execute instructions on variables that expose this information must be adapted or created for every system. This is part of the motivation for DyND and why very soon the datashape system and its C++ type library will be released independently from the rest of DyND and Blaze. This is fundamentally why DyND and datashape are such important projects to me. I see in them the long-term path to massive code-reuse, the breaking down of data-silos that currently cause so much analytics algorithm duplication and lack of cooperation.
Simple algorithms from data-munging scripts to complex machine-learning solutions must currently be re-built for every-kind of data-silo unless there is a common way to actually functionally bring code to data. Datashape and the type-library runtime from DyND (ndt) will allow this future to exist. I am eager to see the Apache Arrow project succeed as well because it has related goals (though more narrowly defined).
The next step in this direction is an on-disk and in-memory data-fabric that allows data to exist in a distributed file-system or a shared-memory across a cluster with a pointer to the head of that data along with a data-shape description of how to interpret that pointer so that any language that can understand the bytes in that layout can be used to execute analytics on those bytes. The C++ type run-time stands ready to support any language that wants to parse and understand data-shape-described pointers in this future data-fabric.
From one point of view, this DyND and data-fabric effort are a natural evolution of the efforts I started in 1998 that led to the creation of SciPy and NumPy. We built a system that allows existing algorithms in C/C++ and Fortran to be applied to any data in Python. The evolution of that effort will allow algorithms from many other languages to be applied to any data in memory across a cluster.
The key part of Blaze that is also important to mention is the notion of the Blaze server and user-facing Blaze expressions and functions. This is now what Blaze the project actually entails --- while other aspects of Blaze have been pushed into their respective projects. Functionally, the Blaze server allows the data-fabric concept on a machine or a cluster of machines to be exposed to the rest of the internet as a data-url (e.g. http://mydomain.com/catalog/datasource/slice). This data-url can then be consumed as a variable in a Blaze expression --- first across entire organizations and then across the world.
This is the truly exciting part of Blaze that would enable all the data in the world to be as accessible as an already-loaded data-frame or array. The logical expressions and transformations you can then write on those data to be your "logical computer" will then be translated at compute time to the actual run-time instructions as determined by the Blaze server which is mediating communication with various backends depending on where the data is actually located. We are realizing this vision on many data-sets and a certain set of expressions already with a growing collection of backends. It is allowing true "write-once and run anywhere" to be applied to data-transformations and queries and eventually data-analytics. Currently, the data-scientists finds herself to be in a situation similar to the assembly programmer in the 1960s who had to know what machine the code would run on before writing the code. Before beginning a data analytics task, you have to determine which data-silo the data is located in before tackling the task. SQL has provided a database-agnostic layer for years, but it is too limiting for advanced analytics --- and user-defined functions are still database specific.
Continuum's support of blaze development is currently taking place as defined by our consulting customers as well as by the demands of our Anaconda platform and the feature-set of an exciting new product for the Anaconda Platform that will be discussed in the coming weeks and months. This new product will provide a simplified graphical user-experience on top of Blaze expressions, and Bokeh visualizations for rapidly connecting quantitative analysts to their data and allowing explorations that retain provenance and governance. General availability is currently planned for August.
Blaze also spawned additional efforts around fast compressed storage of data (blz which formed the inspiration and initial basis for bcolz) and experiments with castra as well as a popular and straight-forward tool for quickly copying data from one data-silo kind to another (odo).
The most important development to come out of Blaze, however, will have tremendous impact in the short term well before the full Blaze vision is completed. This project is Dask and I'm excited for what Dask will bring to the community in 2016. It is helping us finally deliver on scaled-out NumPy / Pandas and making Anaconda a first-class citizen in Hadoop.
In 2014, Matthew Rocklin started working at Continuum on the Blaze team. Matthew is the well-known author of many functional tools for Python. He has a great blog you should read regularly. His first contribution to Blaze was to adapt a multiple-dispatch system he had built which formed the foundation of both odo and Blaze. He also worked with Andy Terrel and Phillip Cloud to clarify the Blaze library as a front-end to multiple backends like Spark, Impala, Mongo, and NumPy/Pandas.
With these steps taken, it was clear that the Blaze project needed its own first-class backend as well something that the community could rally around to ensure that Python remained a first-class participant in the scale-out conversation --- especially where systems that connected with Hadoop were being promoted. Python should not ultimately be relegated to being a mere front-end system that scripts Spark or Hadoop --- unable to talk directly to the underlying data. This is not how Python achieved its place as a de-facto data-science language. Python should be able to access and execute on the data directly inside Hadoop.
Getting there took time. The first version of dask was released in early 2015 and while distributed work-flows were envisioned, the first versions were focused on out-of-core work-flows --- allowing problem-sizes that were too big to fit in memory to be explored with simple pandas-like and numpy-like APIs.
When Matthew showed me his first version of dask, I was excited. I loved three things about it: 1) It was simple and could, therefore, be used as a foundation for parallel PyData. 2) It leveraged already existing code and infrastructure in NumPy and Pandas. 3) It had very clean separation between collections like arrays and data-frames, the directed graph representation, and the schedulers that executed those graphs. This was the missing piece we needed in the Blaze ecosystem. I immediately directed people on the Blaze team to work with Matt Rocklin on Dask and asked Matt to work full-time on it.
He and the team made great progress and by summer of 2015 had a very nice out-of-core system working with two functioning parallel-schedulers (multi-processing and multi-threaded). There was also a "synchronous" scheduler that could be used for debugging the graph and the system showed well enough throughout 2015 to start to be adopted by other projects (scikit-image and xarray).
In the summer of 2015, Matt began working on the distributed scheduler. By fall of 2015, he had a very nice core system leveraging the hard work of the Python community. He built the API around the concepts of asynchronous computing already being promoted in Python 3 (futures) and built dask.distributed on top of tornado. The next several months were spent improving the scheduler by exposing it to as many work-flows as possible from computational-science, quantitative-science and computational-science. By February of 2016, the system was ready to be used by a variety of people interested in distributed computing with Python. This process continues today.
Using dask.dataframes and dask.arrays you can quickly build array- and table-based work-flows with a Pandas-like and NumPy-like syntax respectively that works on data sitting across a cluster.
Anaconda and the PyData ecosystem now had another solution for the scale-out problem --- one whose design and implementation was something I felt could be a default run-time backend for Blaze. As a result, I could get motivated to support, market, and seek additional funding for this effort. Continuum has received some DARPA funding under the XDATA program. However, this money was now spread pretty thin among Bokeh, Numba, Blaze, and now Dask.
With the distributed scheduler basically working and beginning to improve, two problems remained with respect to Hadoop interoperability: 1) direct access to the data sitting in HDFS and 2) interaction with the resource schedulers running most Hadoop clusters (YARN or mesos).
To see how important the next developments are, it is useful to describe an anecdote from early on in our XDATA experience. In the summer of 2013, when the DARPA XDATA program first kicked-off, the program organizers had reserved a large Hadoop cluster (which even had GPUs on some of the nodes). They loaded many data sets onto the cluster and communicated about its existence to all of the teams who had gathered to collaborate on getting insights out of "Big Data." However, a large number of the people collaborating were using Python, R, or C++. To them the Hadoop cluster was inaccessible as there was very little they could use to interact with the data stored in HDFS (beyond some high-latency and low-bandwidth streaming approaches) and nothing they could do to interact with the scheduler directly (without writing Scala or Java code). The Hadoop cluster sat idle for most of the summer while teams scrambled to get their own hardware to run their code on and deliver their results.
This same situation we encountered in 2013 exists in many organizations today. People have large Hadoop infrastructures, but are not connecting that infrastructure effectively to their data-scientists who are more comfortable in Python, R, or some-other high-level (non JVM language).
With dask working reasonably well, tackling this data-connection problem head on became an important part of our Anaconda for Hadoop story and so in December of 2015 we began two initiatives to connect Anaconda directly to Hadoop. Getting data from HDFS turned out to be much easier than we had initially expected because of the hard-work of many others. There had been quite a bit of work building a C++ interface to Hadoop at Pivotal that had culminated in a library called libhdfs3. Continuum wrote a Python interface to that library quickly, and it now exists as the hdfs3 library under the Dask organization on Github.
The second project was a little more involved as we needed to integrate with YARN directly. Continuum developers worked on this and produced a Python library that communicates directly to the YARN classes (using Scala) in order to allow the Python developer to control computing resources as well as spread files to the Hadoop cluster. This project is called knit, and we expect to connect it to mesos and other cluster resource managers in the near future (if you would like to sponsor this effort, please get in touch with me).
Early releases of hdfs3 and knit were available by the end of February 2015. At that time, these projects were joined with dask.distributed and the dask code-base into a new Github organization called Dask. The graduation of Dask into its own organization signified an important milestone that dask was now ready for rapid improvement and growth alongside Spark as a first-class execution engine in the Hadoop ecosystem.
Our initial goals for Dask are to build enough examples, capability, and awareness so that every PySpark user tries Dask to see if it helps them. We also want Dask to be a compatible and respected member of the growing Hadoop execution-framework community. We are also seeking to enable Dask to be used by scientists of all kinds who have both array and table data stored on central file-systems and distributed file-systems outside of the Hadoop ecosystem.
With Dask (including hdfs3 and knit), Anaconda is now able to participate on an equal footing with every other execution framework for Hadoop. Because of the vast reaches of Anaconda Python and Anaconda R communities, this means that a lot of native code can now be integrated to Hadoop much more easily, and any company that has stored their data in HDFS or other distributed file system (like s3fs or gpfs) can now connect that data easily to the entire Python and/or R computing stack.
This is exciting news! While we are cautious because these integrative technologies are still young, they are connected to and leveraging the very mature PyData ecosystem. While benchmarks can be misleading, we have a few benchmarks that I believe accurately reflect the reality of what parallel and distributed Anaconda can do and how it relates to other Hadoop systems. For array-based and table-based computing workflows, Dask will be 10x to 100x faster than an equivalent PySpark solution. For applications where you are not using arrays or tables (i.e. word-count using a dask.bag), Dask is a little bit slower than a similar PySpark solution. However, I would argue that Dask is much more Pythonic and easier to understand for someone who has learned Python.
It will be very interesting to see what the next year brings as more and more people realize what is now available to them in Anaconda. The PyData crowd will now have instant access to cluster computing at a scale that has previously been accessible only by learning complicated new systems based on the JVM or paying an unfortunate performance penalty. The Hadoop crowd will now have direct and optimized access to entire classes of algorithms from Python (and R) that they have not previously been used to.
It will take time for this news and these new capabilities to percolate, be tested, and find use-cases that resonate with the particular problems people actually encounter in practice. I look forward to helping many of you take the leap into using Anaconda at scale in 2016.
We will be showing off aspects of the new technology at Strata in San Jose in the Continuum booth #1336 (look for Anaconda logo and mark). We have already announced at a high-level some of the capabilities: Peter and I will both be at Strata along with several of the talented people at Continuum. If you are attending drop by and say hello.
We first came to Strata on behalf of Continuum in 2012 in Santa Clara. We announced that we were going to bring you scaled-out NumPy. We are now beginning to deliver on this promise with Dask. We brought you scaled-up NumPy with Numba. Blaze and Bokeh will continue to bring them together along with the rest of the larger data community to provide real insight on data --- where-ever it is stored. Try out Dask and join the new scaled-out PyData story which is richer than ever before, has a larger community than ever before, and has a brighter future than ever before.
suitable tools for my (admittedly complex) task.
installed onto many users machines.
days building their NumPy stack and literally become fearful of changing it.
Linux users who had apt-get and yum).
package managers exist --- on Windows things like EPD, Anaconda or C.
Gohlke's collection of binaries have been the only solution.
of how different compilers and run-times define the `FILE *` file pointer.
developers. I'm grateful Dave Beazley recommended it to me over 6 years ago.
a "standard" that was actually limiting for a sizeable set of users.
environment needed before running any code to do the actual build.
have. Anything else is going to require all kinds of "bootstrap"
require that all things begin with the python setup.py incantation.
can support any kind of binary file. Making conda packages is as simple as making a recipe for it. We make a growing collection of public-domain, example recipes available to everyone and also encourage attachment of a conda recipe directory to every project that needs binaries.
At the heart of conda package installation is the concept of environments.
default (root) environment if you don't explicitly specify another one.
quickly as we can identify them.
"binstar in beta") which allows you to host your own binary conda packages.
people. In this context, Anaconda could be considered a "reference"
distribution and a proof of concept of how to use the conda package manager.
Wakari also uses the conda package manager at its core to share bundles.
create this distribution pretty easily and compete with us.
from others in the community).
solution for everyone: users, developers, and IT administrators alike.
with anything else in the Python packaging space. I do believe that the group of people working on Python packaging and Nick Coghlan in particular are doing a valuable service. It's a very difficult job to take into account the history of Python packaging, fix all the little issues around it, *and* provide a binary distribution system that allows users to not have to think about packaging and distribution. With our resources we did just the latter. I admire those who are on the front lines of the former and look to provide as much context as I can to ensure that any future decisions take our use-cases into account. I am looking forward to continuing to work with the community to reach future solutions that benefit everyone.
I attended a few days of SciPy 2013 and enjoyed interacting with the many old friends and many new friends that participate in this conference. I thought the program committee did an excellent job of selecting talks and there were more attendees this year which also mirrors my experience with the PyData conference series which sells out every time. Andy Terrell, a NumFOCUS board member and researcher at the University of Texas, and Jonathan Rocher, an Enthought developer, were co-chairs of SciPy this year and did an excellent job of coordination.
Continuum Analytics, my new company, is the institutional sponsor of the PyData conference series and I know how much work it can be, so my thanks go out to Enthought for their efforts to sponsor the SciPy conference this year and in years past. I'm really looking forward to the day when the SciPy conference, like the PyData conference series, directly benefits NumFOCUS which is a non-profit organization with 501(c)(3) status started by the scientific Python community and run by the same community behind so much of the SciPy stack. It looks like steps are being taken in that direction which is wonderful to see. At the SciPy conference, Fernando Perez, of IPython fame, led the charge to get fiscal sponsorship documents improved to make it much simpler for people wanting to sponsor the great projects on the scientific python stack (IPython, NumPy, SciPy, Pandas, SymPy, Matplotlib, etc.) to have a vehicle to do it. This year, NumFOCUS was able to sponsor the attendance of two students to the SciPy conference because of generous donors. Right now, NumFOCUS is looking for help for its website to improve the look and feel. It's a great way to get involved with the community and help out. Just send an email to the numfocus google group (a public group for all to get involved with): mailto:numfocus+subscribe@googlegroups.com?subject=Subscribe.
Right now, a conversation involving graph-representations for Python compilation tools is happening on the numfocus mailing list among several interested parties from SymPy, Numba, Theano, Pythran, Parakeet, etc. One of the highlights of the conference for me was meeting and interacting with other people interested in Python-for-science compiler technology as it looks like there is a healthy community developing around this topic. I hope those interested in the topic check out compilers.pydata.org and issue pull requests to that github-hosted page to describe their favorite tool.
I only attended some of the tutorial given by fellow Continuum team members Ben Zaitlen and Clayton Davis. I was gratified to see that wakari.io was useful for so many people during the tutorials, and appreciated the feedback on how we can continue to improve the tool. I'm also grateful to see all the people able to productively use Anaconda which is our free, cross-platform, distribution for using Python for scientific work and data analysis.
I really enjoyed the talk on Spyder by Carlos Cordoba. The Spyder IDE is a very nice tool and I was happy to see Carlos promoting it. The Spyder IDE is featured in our Anaconda Launcher (part of the Anaconda 1.6 release) along with the IPython notebook and IPython console. The Launcher allows anyone to publish their app to multiple platforms simply by making a conda package (with an icon and an entry-point) and upload it to a repository that the Launcher is looking at. All the dependencies can be specified and they will be installed via conda automatically when the app is selected. The hope is to make it very easy for anyone to get their cool application based on Python in front of people quickly without having to make installers for every platform.
Besides the excellent keynote talks, by Fernando Perez, William Schroeder, and Olivier Grisel, I also found the talks by Matthew Rocklin, Pat Marion, Ramalingam Saravanan, Serge Guelton, Samuel Skillman, Jake Vanderplas, and Joshua Warner very interesting. It was especially nice to meet Joshua who was coming from the Mayo Clinic where SciPy began. I started writing the SciPy library in 1999 at the Mayo Clinic while I was a graduate student there (then called Multipack, special, and a bunch of other modules). It was very nice to meet someone from Mayo contributing again to this community with a very nice fuzzy logic package based on the work of an old professor of mine Hal Otteson. His work is now a new scikit. The scikit concept has been a tremendous boon for development of the Scientific Python community as it allows more distributed development and more rapid expansion of the available tools. If better packaging had existed at the time, I would very likely have kept my early modules independent so they could grow with their own developer bases. What is now the SciPy library should most likely have been a SciPy distribution (with perhaps a smaller core). But, hindsight is 20/20 and given the state of the world at the time, the best option seemed to be to create the SciPy library with Eric Jones and Pearu Peterson.
Mark Wiebe did an excellent job in presenting dynd, a C++ library for dynamic multi-dimensional array manipulation with nice python bindings. Mark's work, sponsored by Continuum Analytics, is something that could lead to NumPy 2.0, although nobody has suggested exactly how that might work yet. As dynd forms a foundation for Blaze, and Blaze and NumPy can co-exist for many years, I haven't been thinking much about how NumPy 2.0 could grow out of dynd until now. I do now have some ideas about how NumPy could be improved that I think will help the space evolve more fluidly and productively with many interested people able to coordinate their varied efforts. The most important of these is the introduction of multi-methods into NumPy which I'll outline below.
I participated on a panel about the future of Array Oriented Computing in Python. Of course, I've been spending a lot of time over the past year working and thinking exactly about that, so I would have preferred a talk versus a panel with only a limited amount of time. However, I have limited time to prepare talks and will be speaking at the upcoming PyData conference in Boston, so I was grateful for the chance to at least express some of the ideas we've been working on. To be clear, I think that Blaze is the future of Array Oriented Computing in Python, though we have some work ahead to prove that out. Exactly what the transition from NumPy to Blaze looks like for people will be a story I care quite a bit about and will be telling more and more in the coming months and years. I take personal responsibility for anyone who adopted NumPy, and I will do everything I can to make sure their transition to using Blaze is as simple as possible. Backward compatibility is very important to me. I spent many hours making sure that NumPy was compatible with both Numarray and Numeric. Fortunately, Blaze and NumPy can co-exist and so there is less of a story of either / or and more about which / when (especially during the transition phase).
There is also another possibility that will be interesting to see if it emerges: retro-fitting NumPy with multi-methods (dispatching on python type and also on dtype). I think this is the single-most important thing that can be done for NumPy. If someone is motivated and has budget, I can work with her to do this in about 1-2 months (maybe even sooner depending on the experience). This is not on my immediately funded road-map, however, so it would need outside funding and/or interest.
There are several different multi-method implementations for Python. For those unfamiliar with the concept, here is a good essay by Guido on the general concept. Multi-methods are also at the heart of Julia. They are a simple concept. Basically, a multi-method is an object that dispatches to a different implementation based on the number and types of the arguments. The idea is that you can add new implementations of the underlying function quite easily without changing the function object itself. So, for example, if numpy.dot were a multi-method, then I could change the implementation of numpy.dot for my new fancy array-object without directly changing the source-code of numpy.dot in NumPy and all downstream functions and methods that use numpy.dot in their implementation would automatically work with my new type of array. Multi-methods allow extensibility in a manner similar to how operator overloading allows extensibility in object-oriented programming. But, it's a much more natural fit for operations where dispatching only on the first argument does not make a lot of sense.
In fact, at the heart of NumPy's ufuncs is a multi-method dispatch mechanism (on NumPy dtype, instead of Python type), so NumPy users have been using multi-methods for a long time. In fact, if NumPy's ufuncs were true multi-methods to begin with, then all the hassle with __array_wrap__, __array_prepare__, and so forth which are hacks to compensate for the lack of true Python-type-based multi-methods would not be necessary. If you look at the implementation of NumPy's masked array's for example you will see some of the ugliness that is caused by NumPy's lack of a better multi-method mechanism. Numba's autojit also effectively creates a kind of multi-method as it creates a new function to dispatch to whenever it encounters a new set of types for the arguments. These are the ideas that we are building on and using in Blaze, as we learn from our experience with NumPy.
The biggest challenge for multi-methods is always what function to return if you don't find an exact match. A simple multi-method is basically a dictionary whose key is the a tuple of the types of the input arguments and whose value is the implementation. But, what do you do if the key does not return an implementation? How do you find a compatible function and use it instead? There is a lot of theory on this and several approaches people have taken. I'm not aware of a universal solution that everybody agrees should be used. However, there are reasonable approaches that can be taken using the idea of typesets or type-hierarchies (for those interested you can read more about contravariance and covariance for other approaches to resolving the type dispatch problem as well).
I'm confident that useful if not universal approaches to this problem can be found (several are already available for Python and in Julia, for example). For NumPy, what is needed is a two-tiered dispatch mechanism. My view is that all NumPy (and SciPy and Scikit) functions should be multi-methods that dispatch based on Python-type *and* then additionally for memory-view-like objects on the data-type of the elements. The dispatch rules for each of these cases can and should be separate, I think.
If you are interested in this problem and especially if you have money to fund it, feel free to contact me directly at travis at continuum dot io.
While I am spending more and more of my conference time with the PyData conference series, I still enjoy reconnecting with people I will always consider friends at the SciPy conference. Fortunately, many speakers participate in both. Having both conferences allows the community to grow and have bigger and better impact as I think can be witnessed by the increased attendance this year at SciPy.


text:
Skip to main content World Africa Americas Asia Europe Middle East Foreign Correspondents
India's corruption scandals spawn citizen protest
By Rama Lakshmi
Rama Lakshmi
Bio Follow
NEW DELHI — Indians fed up with recent corruption scandals have launched a nationwide protest demanding that they be allowed to participate in drafting a tough, national anti-graft law.
The multi-city campaign called "India Against Corruption" is aimed at stopping the proposed Ombudsman Bill, which critics say is too weak. It allows a government ombudsman to recommend cases for investigation but does not give the ombudsman any police-like power.
A coalition of social crusaders across India have drafted their own parallel and more stringent bill, called the Citizens' Ombudsman Bill, which would empower an ombudsman to receive complaints directly from citizens and investigate and prosecute them.
The coalition wants citizen input into the selection of an ombudsman, and is calling for the ombudsman to have the power to pursue charges against judges and the prime minister's office.
"We will not accept another weak, toothless and merely advisory bill. This is hopeless. We do not have any faith in the government's bill," said Kiran Bedi, a retired police officer and social worker who was among hundreds protesting in New Delhi on Wednesday.
"Is there no political will to end corruption in this country?"
The demonstrators — students, teachers, executives, lawyers, doctors, retired officials and activists — held placards and shouted slogans denouncing the government-proposed ombudsman's bill, which was drafted 42 years ago and has been introduced 10 times in parliament but never passed.
Anna Hazare, a 72-year-old activist, began a hunger strike Tuesday demanding that the government consult citizens before finalizing the bill.
"The government's proposal is mere eyewash," said Hazare, sitting under a white tent and flanked by protesters on the second day of his fast. "It does not strengthen the anti-corruption fight in India. All that we are asking the government to do is to form a joint committee, consult us, allow the citizens to participate in the drafting of a new anti-corruption bill."
The campaign's online petition has logged about 570,000 supporters.
Officials say they are open to suggestions from the people, but they insist that people cannot write legislation.
Manish Tewari, a Congress Party spokesman, said that Hazare's fast was "not appropriate, probably unnecessary and premature, because the government has already formed a panel" to examine the contours of the anti-corruption legislation.
But India's law minister, Veerappa Moily, told reporters that the government is open for more discussion. "We have not closed the issue. But we said that we will not be able to meet now, for constraint of time and the pressure of elections," Moily told reporters in his office. "The bill has gone through the mill for 42 years. All right, if you want more discussions, then we are prepared to reflect."
In the past six months, Prime Minister Manmohan Singh's government has been rocked by a series of big-ticket corruption scandals that have triggered massive public outrage. They include a mammoth $39 billion telecom scandal, cutbacks in organizing last year's Commonwealth Games, and the grabbing by politicians and bureaucrats of Mumbai apartments meant for war widows and veterans.
On Tuesday, Anil Ambani, one of India's leading business tycoons, testified before a parliamentary panel investigating the telecom scandal. Police have charged employees of one of his companies, Reliance Telecom, with cheating to secure telecom licenses from the government.
"There is no limit to corruption now in this country. This is a movement of people's anger. We have lost patience," said Minu Kothari, 20, a university student and volunteer at the protest site in New Delhi.
Singh's Congress party faces elections in five key states this month; many observers consider those votes a midterm referendum on his rule.
"I have written so many letters to the prime minister asking for time to meet and discuss. No reply," Hazare told reporters. "Everybody is drowned in corruption. I will fast until I die for the cause of cleaning up the system."


text:
Jeep Renegade vs Wrangler
January 6, 2023 by Mario Ryan
Of all the Jeep vehicles, the Jeep Renegade and the Jeep Wrangler are the two SUVs that are often compared because of their abilities to take you on one-of-a-kind adventures. These two models share a lot of similar features but also have some differences that need to be discussed to compare the two.
So, what are the differences between the Jeep Renegade vs Wrangler? This article differentiates the two Jeep models in terms of pricing, physical features, and other factors to allow Jeep enthusiasts to choose the best vehicle for their needs.
Point of Difference Jeep Wrangler Jeep Renegade
Pricing More Expensive Less Expensive
Size Smaller Bigger
Physical Appearance Rugged Style Modern Look
Driving Capability Efficiency Off-Road Capability
Size and Space
Towing and Cargo Capacity
Overall Difference
The easiest way to compare these two Jeep models is the price. Jeep Renegade is much cheaper and more affordable in pricing compared to the cost of the Jeep Wrangler. The Jeep Renegade is around $18,000, while the Jeep Wrangler is around $ 24,000. That is a $6,000 difference between these two vehicles.
Passenger Volume 88.4 Cubic Feet 118.6 Cubic Feet
Cargo Space 56.5 Cubic Feet 50.8 Cubic Feet
The vehicle's size and space is also an excellent indicator to differentiate Jeep Renegade vs Jeep Wrangler. To be able to know exactly the difference in size and space, we must take into account the passenger volume and the cargo space measurement of these two vehicles.
As for the passenger volume, the Jeep Renegade is relatively bigger at 118.6 cubic feet. The Jeep Wrangler is smaller as its passenger volume only measures 88.4 cubic feet. On the other hand, the Jeep Wrangler has a large cargo space between the two vehicles at 56.5 cubic feet. The Jeep Renegade only measures 50.8 cubic feet.
Engine 3.6 Liter V6 2.4 Liter 4 Cylinder1.4 Liter Turbo
4×4 Systems Command-TracRock-Trac Jeep Active DriveJeep Active Drive Low
Heavy-Duty Axle
Tru-Lok Locking Differentials
Sway Bar Disconnect System
2WD System
Selec-TrainTaction Control System
Taking a look at all the features offered in these two models will also make them unique from each other. In terms of vehicle features, the Jeep Wrangler is powered by a 3.6 Liter V6 engine which offers 285 horsepower and 21 highway MPG.
On the other hand, the Jeep Renegade offers two options for engines. The first engine is the 2.4 liter 4 Cylinder that provides 180 horsepower and 30 highway MPG. The other engine option is the 1.4-liter turbo that delivers 160 horsepower and 31 highway MPG.
4×4 Systems
The Jeep Renegade and the Jeep Wrangler offer two different 4×4 systems. The Jeep Wrangler offers two options: Command-Trac or Rock-Trac. On the other hand, the Jeep Renegade provides Jeep Active Drive or the Jeep Active Drive Low.
When it comes to the other features, the Jeep Renegade has an available 2WD system. It also has a Selec-Terrain Traction Control System, which enables the Jeep Renegade to adjust to different types of terrain.
Meanwhile, the Jeep Wrangler offers various features, such as a heavy-duty axle. It also comes with a Tru-Lok locking differential and a sway bar disconnect system which promotes optimal road performance.
Towing Capacity 2000 – 3500 lbs 2000 lbs
Cargo Capacity 72.4 Cubic Feet 50.8 Cubic Feet
The current editions of the Jeep Renegade and the Jeep Wrangler can also be differentiated by their towing and cargo capacity. The 2022 edition of the Jeep Wrangler is much stronger and can tow from 2000 to 3000 lbs. On the other hand, the Jeep Renegade can tow up to 2000 lbs.
On the other hand, the two vehicles' cargo capacity is also different in terms of sizing. The Jeep Wrangler's cargo can accommodate up to 72.4 cubic feet of cargo, while the Jeep Renegade can only accommodate up to 50.8 cubic feet.
As shown in the article above, both vehicles have plenty to offer in terms of features. But, if you are looking for a vehicle that offers efficiency during the ride, the Jeep Renegade is the best bet. If you are looking for more off-road capability, then the Jeep Wrangler is the best choice.
Meanwhile, if you love a vehicle's modern look and an affordable price tag, you can definitely consider the Jeep Renegade. On the other hand, take the Jeep Wrangler for more power and a rugged look for a vehicle.
Comparing Jeep Renegade vs Wrangler is easy since there are a lot of differences between the two vehicles. But if you want to determine which vehicle is the right one, take each Jeep model on a test drive and see the difference. Our team hopes that the information provided in this article helps you differentiate these two vehicles and guide you to choose the right one for your needs.
How to Sell my Jeep
How Fast Can a Jeep Wrangler Go