This release of the Python agent includes various improvements and bug fixes related to instrumentation for database client modules, as well as a notable fix to our 'newrelic-admin' script affecting some users who referenced Python virtual environments via a symbolic link.
Instrumentation of the WSGI application entry point will now add a new breakdown metric to web transactions corresponding to the finalization of the WSGI request. Within the workings of the interaction between the WSGI server and the WSGI application, this is the point at which the WSGI server will call any close() method on the iterable returned by the WSGI application. The name of this new breakdown metric is 'WSGI/Finalize'. In the case where there was actually a 'close()' method, a further breakdown metric will also appear corresponding to that method.
Added support to instrumentation for MySQL and PostgreSQL database client modules for monitoring database queries when the database connection object is used as a context manager. Such context manager features are outside of the scope of the Python DBAPI2 (PEP 249) specification, not all database client modules support it and nor do they all work the same. Although supported, we would suggest consideration should be given to not using these context manager features if you need your code to be portable between databases.
Added support to instrumentation for MySQL database client modules for monitoring database queries when the database connection object was created using the Connect() function. The Connect() function falls outside of the scope of the Python DBAPI2 (PEP 249) specification. Although supported, we would suggest consideration should be given to not using this Connect() function if you need your code to be portable between databases.
Added database instrumentation support for the mysql-connector-python database client module.
Custom parameters for a transaction which are a string or numeric value will now be added to and reported with analytic events reported to our analytics system code named Rubicon. This can be disabled using the agent configuration setting 'analytics_events.capture_attributes'.
The capture of custom parameters against a transaction trace can now be disabled using the agent configuration setting 'transaction_tracer.capture_attributes'.
The capture of custom parameters against error details can now be disabled using the agent configuration setting 'error_collector.capture_attributes'.
The agent API function add_user_attribute() is now deprecated and functionality merged with the add_custom_parameter() function. The latter function should now be used instead. The display of such parameters in browser traces is now optionally enabled with the agent configuration setting 'browser_monitoring.capture_attributes'.
If the 'newrelic' package was installed into a Python virtual environment, but the 'newrelic-admin' script was executed via a path that traversed a symlink to the virtual environment, the protections within the agent bootstrapping procedure was detecting that the agent was trying to be used with an application running against a different virtual environment when it was actually the same. This would result in the application not being monitored. This issue was introduced in version 2.8.0 of the agent when additional protections were added against mixing application/modules from different Python virtual environments.
Explain plans were not being performed on SQL queries made via the executemany() method of a database cursor object. When explain plans are now done, the data inputs from the first row of input data for the executemany() call will be used.
When using Python 2, if strings were supplied for the web transaction name, custom parameters, in error details etc, and that string contained a series of characters which could not be decoded as valid UTF-8, then an exception would occur. In the case of a web transaction name, this could result in the exception affecting the current web transaction and result in an error response being sent back to a user. For the case of a transaction trace or error details, the exception would prevent the sending of the captured data up to our data collector and it would be discarded. This was a regression within the agent behaviour introduced when Python 3 support was added to the agent.
Fixed instrumentation for sqlite database modules which could result in instrumentation not being applied correctly, and so no database metrics collected, if the sqlite module had been imported prior to the agent being initialized.
Limits being applied to the length of the SQL for a slow SQL query when being sent up to our data collector were being applied at the wrong time, resulting in the truncated SQL being used when performing an explain plan. This didn't affect the operation of the web application, but database logs could contain an error about the malformed SQL query.
Explain plans could be attempted for an SQL query even where the SQL query failed. Under most circumstances a SQL query would fail immediately and so the duration would fall below the threshold for collecting an explain plan, but the changes now made will protect against a long running SQL query which failed in the database and ensure that no additional problem is caused by issuing an explain plan for it.
If a monitored web application is started up using our newrelic-admin wrapper script, and it executes a separate Python script and that Python script used a Python version older than Python 2.6, the script could output the error message "'import site' failed; use -v for traceback". The execution of the script was not affected, but the message obviously could cause concern.
The equivalent functions from the 'urllib2' module from Python 2 were not being instrumented when Python 3 was being used. This was missed when Python 3 support was added to the agent.
This release of the Python agent is a hot fix version to address a Celery instrumentation issue introduced with version 2.2.0.2.
Fixed bug with Celery instrumentation introduced with version 2.2.0.2 of the agent. The bug was causing a Python exception to be raised when the instrumentation was trying to derive the task name to use as the name of the background task.
This release of the Python agent adds support for Cross Application Tracing.
Support for New Relic's Cross Application Tracing feature is now available for the Python agent. This feature enables correlation of transaction traces between different services in your stack. It works on external calls made between applications monitored by any New Relic agents (Java, .NET, Ruby and Python), which support this feature. At this time the Python agent supports this feature on outgoing calls when using the 'httplib' module or any other module which uses 'httplib', such as the 'urllib', 'urllib2', 'urllib3' and 'requests' modules. Any inbound calls from other agents which support the feature will also be handled.
Added instrumentation for the weberror package. This provides visibility into potentially blocking operations such as sending of email for exceptions which are being reported.
Added instrumentation for the umemcache package.
Added instrumentation for the DBAPI2 compliant interface for the IBM DB2 ibm_db_dbi database client package.
When running a WSGI application under the Tornado WSGI container, data reporting would stop if an unhandled exception managed to propagate back up to the Tornado WSGI container. This was due to a bug in Tornado in respect of its compliance with the WSGI specification. A workaround is provided to avoid the problems this caused to the Python agent. Tornado versions prior to 3.2 will still carry that bug however and we make no attempt to address the bug in Tornado.
The Tornado instrumentation was causing template rendering to fail where a relative path was used to refer to the template and no template path had been specified for the RequestHandler instance or globally.
The Tornado instrumentation was causing exceptions when using the @tornado.gen decorators under Tornado 2.X.
When using Django, web transactions will now be named after individual view handlers when using class based views, rather than being named after the class itself.
When using Pyramid, web transactions will now be named after individual view handlers when using view classes, rather than being named after the class itself.
Celery instrumentation had stopped working correctly for Celery versions 2.5.3 through 2.5.5.
No data was being reported where a monitored process was being shutdown within a couple of seconds of being started even if the agent was able to register. This could result in custom metrics in particular not being reported.
SSL certificate validation was failing on older Debian systems due to the OpenSSL libraries not being able to process a couple of the certificates bundled with the Python agent. Those certificates were not required and have been removed.
Calculation of a time string was failing on Windows due to using a strftime() formatter which only existed on UNIX systems.
This release of the Python agent introduces a major new feature called X-Ray Sessions. This feature allows you to gain deeper insights into a key transaction's performance by showing transaction traces alongside long-running profiler results.
X-Ray Sessions provide more targeted transaction trace samples and thread profiling for web transactions. For full details see our X-Ray sessions documentation.
For new agent installations, the agent will now default to using a SSL connection when connecting to our data collector to report any data. If updating from an older agent version, you will however need to manually update your agent configuration file and change the 'ssl' setting to 'true' to always use SSL connections.
When connecting direct to our data collector over a SSL connection, SSL certificate validation will now be performed to mitigate the possibility of man in the middle attacks.
Breakdown metrics for transactions are now recorded as unscoped metrics as well as the existing metrics scoped to a web transaction. As custom dashboards can only chart unscoped metrics, what this means is that it is now possible to create custom dashboards that display data on specific breakdown metrics, such as time spent in traced functions. For example, you may wish to chart time spent in creating database connections, or specific middleware and view handler functions.
When displaying error details, the full path of the exception is now displayed and not just the class name. This makes it easier to correctly identify the full name, with that name then being able to be directly copied into the 'error_collector.ignore_errors' setting of the agent configuration file if it is desired that that exception be ignored.
Added the 'newrelic.agent.record_custom_metrics()' function to the public API for the agent to allow the recording of many custom metrics in one function call to the agent.
We have temporarily disabled the generation of CPU burn information for transaction traces. This was displaying inaccurate values after a UI change, but also had an underlying issue whereby it was reporting CPU burn by the whole process and not just the thread handling the request. We are withdrawing the feature until we can improve the accuracy of the reported value.
We have disabled the ability to get thread profiling information when a WSGI server makes use of coroutine libraries such as gevent or eventlet. This is because the nature of how coroutines are implemented, means we were are not able to generate accurate usable data.
When queueing Celery tasks from a web application, those web transactions were in some cases being wrongly categorised as background tasks. The name of the transaction was also being labelled with a string with the object representation, resulting in metric grouping issues.
The older version of the 'requests' package we were bundling and were using for any HTTP calls was not closing socket descriptors immediately when a requests session object was disposed of. This could result in a small number of open socket descriptors being held open until the Python garbage collector kicked in and closed them.
Login credentials supplied as part of a URL, could in some circumstances appear with the URL parameter associated with a web external node in a transaction trace sample.
Django 1.5 moved which module the 'Http404' exception was declared in. As a result we were no longer correctly ignoring it when catching exceptions to process as errors.
A check designed to warn against the use of uWSGI versions prior to 1.2.6, which contain a bug which results in the agent recording incorrect response times, was failing on very old uWSGI versions.
Updated our gevent instrumentation to accommodate a prototype change in gevent 1.0rc1.
Updated our bottle instrumentation to handle the non numeric version string used in bottle development versions.
After five minutes of being unable to connect to our data collector to report data, the agent will now internally reset itself, discarding accumulated metric data and require the agent to reregister itself with our core application before collecting data once again. This is to ensure that we do not unnecessarily keep accruing data and causing an increase in memory usage while we cannot contact our data collector.
Avoid the potential for a web request to deadlock in a situation where a web framework transaction management system made a copy of the object we use to record details about web transactions or background tasks.
This release of the Python agent fixes an issue when floating point values were used in headers used to calculate queueing time.
When a float value was provided in the X-Request-Start or X-Queue-Start headers as the time stamp, it was not being parsed correctly and the fractional part was being discarded. This was resulting in loss off accuracy.
This release of the Python agent contains a fix for when reporting to multiple applications, as well as other minor bug fixes and enhancements.
On the Linux platform, the agent environment now reports the total amount of system memory available.
Added support for psycopg2cffi database module.
Custom parameters can be associated with function trace nodes for display against transaction traces in the UI.
The URL associated with a call to an external service will now be shown in the custom parameters section for the node in a transaction trace. Previously would have been shown as a label when hovering other the node.
The agent was not reporting any data where multiple applications were being listed in in the app_name configuration setting.
Calling register_application() in the context of a module import could cause a temporary deadlock when a timeout period was specified. The potential for deadlock will now be detected and the function will return without waiting.
When the agent was configured to capture query string parameters, a key with an empty value was being discarded and not reported.
When using gevent mode of gunicorn, the SystemExit exception was being logged on process shutdown due to gevent sending the SystemExit exception to all greenlets and not just the main thread when sys.exit() was called.
Instrumentation wrapping the Django reverse() function was not accepting named parameters.
If custom parameters were used with a transaction, the HTTP response STATUS was duplicated in details related to an error raised from that transaction.
Changes to Django instrumentation to avoid metric grouping issues resulting from raw URLs leaking through as transaction name.
This release of the Python agent focuses on providing a low-overhead thread profiler which can monitor and record executing threads for a specified time, without affecting the customer experience.
Low-overhead thread profiler. A profiling session can be initiated for a period from 2 to 10 minutes. The profiler will periodically sample what all the executing threads are doing, aggregating the results together to get a statistical snapshot of where in your application most time is being spent.
Importing of the greenlet module was resulting in the capacity analysis report metrics being disabled, even though it was meant to only be disabled when specifically using a coroutine based WSGI server such as gevent and eventlet mode of gunicorn.
Improvements to the instrumentation wrapper applied to the Django view handler function so that code which tries to compare the view handler to an expected value should work. Note that this is dependent upon using an equality test rather than an identity test.
Reverted prior change which stripped the instrumentation wrapper from the Django view handler when it was passed to any Django view middleware. Improvements to the instrumentation wrapper should remove the need to do this. Having the view handler passed in with the wrapper applied is required to ensure that calls to the view handler are tracked properly where the view handler is called from within the view middleware.
Although not strictly required due to improvements to the instrumentation wrapper applied to a Django view handler, strip the wrapper when a view handler is passed into django.core.urlresolvers.reverse() function to ensure it can correctly determine the URL mapping to the view handler.
Disable automatic RUM header/footer insertion when using Django 1.5 and the new streaming HTTP response object is returned. This is being disabled so that the response is not buffered, thereby interfering with the streaming process. If streaming a HTML response and RUM reporting is still required, the RUM header and footer will need to be added manually.
Instrumentation of the feedparser module was causing an exception when a file like object rather than a string was passed to the feedparser.parse() function.
Tastypie instrumentation was failing to be applied if using Python 2.5.
This release of the Python agent introduces a number of new features including improvements to RUM browser traces. Performance of the agent has also been improved though reducing some of the overheads when dealing with tracking SQL database queries.
The agent now collects additional application tier transaction trace samples for slow transactions and will correlate them with any corresponding RUM browser trace.
Slow transaction traces will now show a figure corresponding to the amount of CPU burn that occurred during the period the transaction was running. This can be used to get an indication of whether code being executed was CPU intensive or principally waiting on I/O.
Slow transaction traces for web requests will now provide an indication of how many concurrent requests were executing at the same time in the process the request was being handled. This can be used to determine how busy the process was at that time, but would also be used in conjunction with the recorded CPU burn to determine whether CPU burn was all from the one transaction or could also be due to parallel executing web requests in a multithreaded application. Note that this measure is dependent on an optional C extension component of the agent being able to be installed at the time of installation of the agent.
Slow transaction traces for web requests will now display information about the amount of request content read and the amount of response content generated. This will include details about what methods of 'wsgi.input' were used to read content and how long it in total took to read the request content. Similar details are also provided about how response content was generated and how long it took from start to finish.
Slow transaction traces for web requests will now display the value of any front-end queueing time within the server stack which pertained to that specific request. Note that this is dependent on the server stack having being configured to introduce the required header to allow queueing time to be measured. The Python agent recognises the 'X-Queue-Start' header.
Automatic support is now provided for tracking front-end queueing time within the server stack when an application is deployed to Heroku.
An API call and means via the WSGI environ dictionary is now provided for indicating that a transaction should not be a candidate for slow transaction traces. This is to allow one to ignore transactions which are known to always be slow. The API call is 'newrelic.agent.suppress_transaction_trace()'.
Calls to Django APIs for sending emails will now be tracked in the performance breakdown for transactions, as well as being displayed in slow transaction traces.
Overheads within the agent which could affect performance of the application have been reduced. The improvements specifically revolve around the processing of SQL database queries. This is principally to address larger than expected overheads seen by a very small subset of users which had large or more complicated SQL queries. The changes could though still result in a small general reduction in overhead and CPU usage directly attributable to the monitoring down by the agent when SQL database queries are being tracked.
Where it is possible to build C extension components at the time the agent is installed, optional C speedups for JSON will now be installed. This reduces the overhead of JSON encoding when data is being sent by the agent to the New Relic data collector on each one minute harvest cycle. If the C extension cannot be compiled, the agent will fall back to using pure Python code as it has done in prior versions of the agent.
Django instrumentation has been improved and issues addressed with the way that instrumentation wrappers were applied to view handlers. For example, when a view handler is passed to a view middleware, any instrumentation wrapper is now removed. This is because the wrapper could cause issues for some code which tried to do a direct comparison between the view handler passed and an expected value. It is believed this change will fix issues when using Mezzanine.
A sporadic issue with the data collector rejecting uploaded data due to what it believes was malformed JSON should be addressed. This was due to some metric values being aggregated as integers rather than as float values. This resulted in overflow of 64 bit values in the data collector due to Python supporting arbitrary length integers and thus being able to send values larger than 64 bits.
The agent will now no longer fail on import when -OO option is used with the Python interpreter to enable highest level of optimisation. This was an issue as -OO strips documentation strings from code and the agent was not handling the case that the documentation string would then be a None object.


I have some red clay chimney flue I've cut for a small rocket stove to boil maple sap. I have done some searching (I promise I have!)but still have some hesitancies about the type cob to use. I have at my disposal clay, perlite, refractory cement and straw if need be. I had trouble with a large stove that shrunk and cracked when I used perlite:refractory 4:1. I guess my questions would be how many layers, how thick, what materials in what proportions. Thanks in advance.


What Is This: Chinese Stainless Steel Cylinder - Medical? Military?
Thread: What Is This: Chinese Stainless Steel Cylinder - Medical? Military?
Picked this up at auction in a killer lot of old gun sights (Lyman and Redfield).
But this thing is just a mystery. The label is a bit tattered and stained so Google Translate wasn't able to get a great hold on it. From the few words that flashed up I'm confident it's Chinese and I think it's medical.
Measures 5-5/16" tall 3-15/16" in diameter.
The inside appears enameled in a dark brown color.
A Vietnamese gentleman I work with reads a bit of Chinese.
He confirms it's some sort of "powdered" medicine. The 960 is the amount of grams.
The right column, fourth row is some date information - expiration or manufacture.
Can anyone help narrow it down further?
Last edited by rcathey; 09-06-2018 at 10:11 AM.
"...some sort of "powdered" medicine..." Which means it could be anything from bear gall bladder to elephant tusk to dried tiger, um, willy. The lid have any kind of gasket? That'd make the thing water proof.
No gasket. Not anymore anyway.
The threads are very fine - 7 turns or so before it comes off. I'd bet it's watertight anyway. I doubt it's worth anything which is fine...I just need to find something nifty to keep in it. Darn thing's probably indestructible.
3-15/16" diameter isn't much but it'd be a dandy thing for an emergency roll of TP. Or a fire starter kit(ZIP fire starter cubes in foil and some wood strike anywhere matches). Checking for water tight means dunking the thing. Easy low tech test. snicker.
Stuff like that has all kinds of uses. Kind of like coffee cans do.
Chinese 20rd Mag military or police?


Whenever you want to paint a house, decorate the living room or instal new kitchen you might need a little help. Our services include full decorating, wallpapering, floor sanding/vanishing, kitchen installation, interior design and much more. We will work round the clock if need be, also weekends, we will travel to where necessary all over the UK and more. We pride ourselves on the standard of work we do and never shy away from any challenge we get put in front of us. We work in all environments from empty flats to house full with people, we help our customers if they need to move furniture and we work well with other trades.
Let Me Decorate Ltd would welcome the opportunity to earn your trust and deliver you the best service in the industry. We look forward to hearing from you and having a long and successful business relationship.


People are the most important factor for the success of any endeavour.
They develop visions and strategies, guide and teach, manage workflows and operate machines.
People are managers, colleagues and employees, target groups, customers and competitors.
And invariably: people are unique individuals!
Psychology is the science of people, their behaviour at work and in their careers. It describes people's perceptions, thinking, learning and decision-making. Psychologists study emotions and motivations. So while they analyse groups and explore mass phenomena, their special focus is on the individual, the unique person.
Therefore, if it's about people, it's about psychology.
We are psychologists. Our professional expertise and more than 25 years of experience in personnel selection have laid the foundation for our work and inform the development of our techniques.
We have developed an innovative and easy-to-use tool for optimising your human resource decisions: the Ackerschott Basic Cognitions Indicator (abcI).
The abcI captures achievement potential profiles and makes them comparable. Each individual's performance potential and weaknesses can be objectively measured using this scientifically rigorous method. The abcI can be efficiently introduced into your recruiting process and procedures and can also be used for human resource development.
The abcI is an ideal supplement to your selection process.


This article highlights some of the latest UI enhancements added in Apache NiFi 1.5.0.
Additionally, when hovering over the "Go to location" button the tooltip now includes the path of the component.
NiFi 1.5.0 is the first release to integrate with the Apache NiFi Registry. NiFi dataflows can now be versioned on the process group level and easily deployed across different NiFi instances. More information can be found in the HCC article "Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0" and in the "Versioning a Dataflow" section of the NiFi User Guide. However, here are some related UI changes to highlight.
As mentioned previously, more information regarding NiFi and NiFi Registry integration can be found in the "Versioning a Dataflow" section of the NiFi User Guide.


While using a great protectant like Wolfgang Deep Gloss Paint Sealant 3.0 will enhance the look of your paint, the key to a great shine is in the prep work. The protectant is actually the last step to that great looking shine.
Your car's paint is bombarded by contaminants and assaulted by foreign objects every day. These are the things that make your finish look less than perfect and can't be avoided if you want to drive your car. Polishing your car removes defects like swirl, etching, scratches and oxidation while enhancing gloss and preparing your car for the protection it needs and deserves. For all of these types of defects there are different types of polish and pads to use with your Dual Action Polisher.
Step 1) For moderate to severe damage, scratches, heavy swirl and other more serious defects, you would want to use the Orange Light Cutting Lake Country CCS Foam Pad, with Wolfgang Total Swirl Remover 3.0 and put your speed setting at 5 to 6.
Step 2) When it comes to normal swirl, light to moderate oxidation, water etching and light scratches, you would need the White Polishing Lake Country CCS Foam Pad with the Wolfgang Finishing Glaze 3.0 with your speed setting set at 3 to 5.
Step 3) If you don't have much as far as damage but want to enhance gloss and prepare a nice clean surface for your protectant, you would want to use the Green Polishing/Finishing Lake Country CCS Foam Pad and Wolfgang Paintwork Polish Enhancer with your speed set at 1 to 3.
Although you can polish your paint by hand, all of these steps are most effective when you use a dual-action polisher like the Rupes LHR 12E Duetto Random Orbital Polisher or FLEX XC3401 VRG Orbital Polisher because it can do the work your hand is simply not able to do.
Don't forget, polish does not protect. It only improves the look. When you are finished, you will need to protect your work with a protectant or wax like the Wolfgang Deep Gloss Paint Sealant 3.0.


Economists and Financial Iraqi officials that the decline in the Iraqi dinar exchange rate against foreign currencies recently, was a result of increased demand for the dollar, and led to fluctuations in market transactions, reflected directly on public life, said.
As a result, prices of imported goods rose (food, construction, and electrical) compared with the prices that prevailed two months ago, by up to 10%, while the central bank plans a new currency category of 50 thousand dinars ask, as one of containing the crisis procedures.
And the decline of the Iraqi dinar exchange rate against the dollar since the beginning of last month to record about 1230 dinars to the dollar, while the exchange rate was stable over the past years at 1223 dinars to the dollar.
Said Abdul-Razzaq al-Saadi, an economist and former head of the Securities Commission of Anatolia "The decline in the dinar exchange rate against the dollar and exchange rate instability cause volatility in commercial transactions, and to pay the high prices of goods in the domestic market."
"The lack of clarity in the Iraqi Central Bank's policy on exchange rate stability mechanisms adopted by the Bank led to fluctuations in the market," pointing out that "the central bank needs to inter procedures that explain monetary policy in the country."
For his part, the Iraqi Central Bank announced that "the main reasons that led to the decline in the dinar exchange rate against the dollar is escaped hard currency (foreign) out of the country, and breadth of speculative activity in the foreign exchange market (banking) offices."
Said Majid picture member of the governing body of the central bank in a statement of Anatolia, said that "there are a number of reasons, which leads to lower exchange rates and increased demand for hard currency, despite the increase in supply by the Central Bank of Iraq, including foreign travel requests, and capital flight ".
He continued that "it is very difficult to control the exchange rate by only central bank actions, should interfere with the rest of the government institutions concerned to limit capital flight operations."
As solutions and real-time decided the Central Bank of Iraq, according to my photos "B1193 dollar dinars per dollar in the sale of 18 banks, the banks granting three points per dollar, in order to stabilize the exchange rate."
The Finance Committee in the Iraqi Parliament held that "quantifying the amounts sold by the central bank a day led to a decline in the exchange rates," noting that during the next month, "a new currency may contribute to the decline in demand for the dollar will go."
Said Abdul Jabbar, a member of the Committee for the Creator Anatolia correspondent that "the Central Bank of Iraq was already sells a daily average of $ 400 million, it was sold amounts of dollar recently reduced to $ 75 million a day, and this has led to increased demand for the dollar."
He added that "the new paper currency which is scheduled to put on the market next month will have an important role in reducing the demand for the dollar, and the high price of the Iraqi dinar to average."
It is scheduled that the Central Bank of Iraq poses a new currency category of 50 thousand dinars, to be added to other paper categories, namely 25 thousand, 10 thousand, 5 thousand dinars, and thousand dinars and 250 dinars.


text:
Info/Help
Damaged Book Sales
For Prospective Employees
← Back to Sports & Recreation
Surfing Guide to Southern California
By David K. Stern and William S. Cleary
Select Binding HardcoverPaperback
Surfing, unlike many sports, requires no teams, rules, regulations, scores, or stadiums full of spectators. Surfing instead encapsulates personal triumph, in which the individual measures the growth and limits of his or her own capabilities while riding the face of a wave.
Initially published in 1963, this first ever guidebook to California surfing remains a classic that embodies the essence of SoCal surfing during the Golden Years. In addition to understanding the anatomy of the coastline, get the skinny on private vs. public beaches, weather and wind conditions, water temperature, swell classifications, sea life, and the history of surfing. Accented with over 100 aerial photos, action shots, and maps, Stern and Cleary's witty guide provides precise descriptions of the entire southern coast and essentially everything you need to know before hitting the waves.
Although the surfing scene has changed, Surfing Guide to Southern California remains highly relevant for surfers of today and provides a dose of nostalgia for surfers of yesterday.
Echo Point Books & Media, LLC., 22 Browne Ct. Unit 100, Brattleboro, VT 05301, United States(802) 257-6900 X707sales@echopointbooks.com

redpajama_set_name: RedPajamaCommonCrawl


Stars like Kate Upton took aim at Marciano for his behaviour around models in the wake of the #MeToo and Time's Up anti-harassment movements.
AceShowbiz - Fashion mogul Paul Marciano has resigned as the executive chairman at GUESS following allegations of sexual misconduct.
Stars like Kate Upton took aim at Marciano for his behaviour around models in the wake of the #MeToo and Time's Up anti-harassment movements, and now Guess bosses have paid out settlements to multiple women who have accused him of inappropriate behaviour.
A statement from Guess officials, obtained by The Blast, reads: "Mr. Marciano exercised poor judgment in his communications with models and photographers and in placing himself in situations in which plausible allegations of improper conduct could, and did, arise".
Supermodel Upton recently took to Instagram to slam Guess chiefs for attempting to pull off a "stunt" in their sexual misconduct investigation into her allegations against Marciano.
She accused company co-founder Marciano of harassment and assault, claiming he groped her breasts and made inappropriate comments to her during meetings and photoshoots in 2010 and 2011.
An internal investigation was launched by the company in February, with Guess heads forming a special committee to conduct a formal review into the matter, but The Layover actress is convinced the inquiry is bogus, suggesting bosses were just doing it to save face.
"I just learned that Guess hired Paul Marciano's PERSONAL lawyer to run their internal 'investigation'," Upton wrote in a screenshot note on Twitter on 28 February. "I'm truly disheartened to learn that Guess is treating this abuse of power and sexual misconduct (to multiple women) in such a manner".
Upton called the investigation a "stunt" while announcing she has refused to participate as a witness in the inquiry.
"Until Guess makes a commitment to conduct an unbiased investigation, I have declined to participate," she declared. "I look forward to participating if and when Guess retains a truly independent firm to run the entire investigation".
Marciano previously denied all claims of sexual misconduct against him.


The SiPy is a multi-network (Sigfox, WiFi and BLE) development platform. It is programmable with MicroPython and the Pymakr IDE for fast IoT application development, easy programming in-field and extra resilience with network failover. The best blend of speed to deployment and access to all the new Sigfox networks rolling out across Europe, USA, Africa and India.
The SiPy is the second triple bearer MicroPython enabled micro controller on the market today - perfect enterprise grade IoT platform for your connected Things. With the latest Espressif ESP32 chipset the SiPy offers a perfect combination of power, friendliness and flexibility. Create and connect you things everywhere. Fast.
The SiPy a multi-network (Sigfox, WiFi and BLE) development platform. It is programmable with MicroPython and the Pymakr IDE for fast IoT application development, easy programming in-field and extra resilience with network failover. The best blend of speed to deployment and access to new LPWAN networks rolling out across Europe, USA, Africa and India.
The board ships with or without male headers mounted as seen on the pictures.
NOTE: Using the Sigfox radio without the external antenna can lead to damage of the device and is therefore not recommended.
PLEASE NOTE: The Espressif ESP32 firmware on Pycom modules is continuously evolving and buyers are reminded that regular firmware updates will be made available.
If you have any questions about this product by Pycom, contact us by completing and submitting the form below. If you are looking for a specif part number, please include it with your message.


text:
White House condemns violence at Capitol- McEnany
"We condemn it - the president and this administration - in the strongest possible terms," McEnany told reporters in the White House briefing room on Thursday (January 7).
"It is unacceptable, and those that broke the law should be prosecuted to the fullest extent of the law."
She also said, "Those who are working in this building are working to ensure an orderly transition of power. Now it is time for America to unite."
McEnany left the room without taking questions from reporters.
KAYLEIGH MCENANY: I am here to deliver this message on behalf of the entire White House. Let me be clear. The violence we saw yesterday at our nation's capitol was appalling, reprehensible, and antithetical to the American way. We condemn it-- the president and this administration-- in the strongest possible terms. It is unacceptable, and those that broke the law should be prosecuted to the fullest extent of the law.
What we saw yesterday was a group of violent rioters undermining the legitimate First Amendment rights of the many thousands who came to peacefully have their voices heard in our nation's capital. Those who violently besieged our capitol are the opposite of everything this administration stands for. The core value of our administration is the idea that all citizens have the right to live in safety, peace, and freedom.
Those who are working in this building are working to ensure an orderly transition of power. Now it is time for America to unite, to come together, to reject the violence that we have seen. We are one American people under God. Thank you very much.
- Where's the president?
[INTERPOSING VOICES]
- The president violated the law.
Hume on Biden planning major policy changes for first few days in office
Fox News senior political analyst Brit Hume weighs in on what the challenge is for Republicans to stop sweeping legislation introduced by President-elect Joe Biden and Democrats.
Pat Cowan, a Republican official in west Texas, would rather blow up her party than see it controlled by "weak" Republicans who increasingly are distancing themselves from President Donald Trump since the U.S. Capitol riots he is accused of inciting. "You can't tell those Republicans from the Democrats!" she scoffed in an interview at her home in Levelland, Texas.
Constitutionally-speaking, Chief Justice John Roberts is meant to preside over President Trump's impeachment trial, but he apparently wants out, Politico reports.Multiple Republican and Democratic sources have reportedly told Politico that Roberts is seeking a way to avoid the job because of how things played out when he oversaw Trump's first impeachment trial last year. Roberts, Politico notes, has worked hard to keep the Supreme Court apolitical during his tenure, so he was reportedly displeased that he "became a top target of the left" during the proceedings. "He wants no further part of this," one source told Politico, although there's been no official word from Roberts' camp about what he'll ultimately do.Trump's trial is a bit of a constitutional oddity. On the one hand, it's a presidential impeachment, but on the other hand, the trial will take place after he leaves office, which is why there's a chance Roberts may have some wiggle room. Historically, either the vice president or the longest-serving member of the Senate have taken up the mantle for lower-level impeachments, per Politico. That means Vice President-elect Kamala Harris or Sen. Patrick Leahy (D-Vt.) could be the choice. Read more at Politico.More stories from theweek.com Trump's White House staff and alumni are reportedly using the same excuse to skip his big sendoff 5 more scathing cartoons about Trump's 2nd impeachment Trump tried to act like a mob boss. Instead he's just a thug.
Ted Cruz continues to condemn riots despite deep criticism
KTRK – Houston
Mitch McConnell says attack at the Capitol was 'provoked by the president'
Farewell address: Trump stresses record, condemns Capitol riot, does not name Biden
Cecilia Vega Gets New Role in ABC News Washington Shuffle
Tearful Pelosi says Congress members may be prosecuted as she announces Capitol security review
Against all odds, Biden transition team is staffed and primed for White House debut
Armed Protesters Demonstrate at Virginia State Capitol on 'Lobby Day'
Donald Trump Is on His Way Out of the White House, But Not Our Lives
Trump's 'History Of Violence' Exposed In Conservative Website's Damning Supercut
Trump Campaign Allies Played Key Role in Rally That Started Capitol Attack


text:
There is no Social Security crisis (with video)
July 1, 2011	9:46 AM CDT By Tony Pecinovsky
"There is no Social Security crisis. It can pay all benefits in-full and on-time until 2036," Nancy Altman, co-director of Social Security Works, told the People's World in a recent interview.
"Social Security is about preventing people from becoming poor in the first place," Altman said. (See video below.)
As the manufactured deficit-debate crisis rages in Washington, Social Security, Medicaid, Medicare and other vital public services are being attacked as too expensive.
Republicans claim that these "costly entitlement programs" cannot be sustained. They want to leave senior citizens, the disabled and the poor to fend for themselves.
While unions, community groups, senior citizens and young workers acknowledge that the deficit is a real concern for many Americans, they argue that the real problem is a lack of revenue. They say we should raise taxes on corporations and the rich before discussing any benefit cuts.
Altman linked the attacks on Social Security and similar programs with the recent wave of attacks on unions and public services. It is all part of an "ideological attack" on the "role of government," she said, adding, "We would not have any of these programs without unions." (Text continues below video)
Altman, who has worked for 30 years in the areas of pensions and Social Security, is currently chair of the board of directors of the Pension Rights Center, a nonprofit organization dedicated to the protection of beneficiary rights.
From 1983 to 1989, Altman taught courses on private pensions and Social Security at Harvard Law School. She served as Alan Greenspan's assistant in his position as chairman of the bipartisan commission that developed the 1983 Social Security amendments. From 1977 to 1981, Altman was a legislative assistant to Senator John C. Danforth, R-Mo., advising him on Social Security related issues.
Photo: Tony Pecinovsky/PW
Tony Pecinovsky
Tony Pecinovsky serves as fundraising co-chair for St. Louis Jobs with Justice, is a member of the United Media Guild and delegate to the St. Louis Central Labor Council. He is also the president of the St. Louis Workers' Education Society. His work has been published in the St. Louis Labor Tribune, Alternet, Shelterforce, Political Affairs and Z-Magazine, among other publications.
Budget cuts: Trump fiscal plan is a right-wing wish list
Will Democrats sabotage themselves on day one?
The Republican record on Social Security: Getting worse all the time
Letter Carriers open bargaining with U.S. Postal Service on new pact
17-year-old's death from excessive heat may lead to law protecting workers
Working-class issues move to the forefront in Extreme music
GET PEOPLE'S WORLD UPDATES


text:
CSSP Welcomes National Leader and Civil and Human Rights Activist Dr. Joia Crear-Perry to its Board of Directors
Washington, DC (December 15, 2020)—The Center for the Study of Social Policy (CSSP) has named Dr. Joia Crear-Perry, Founder and President of the National Birth Equity Collaborative (NBEC), to its Board of Directors. Dr. Crear-Perry brings to CSSP her knowledge and experience in promoting maternal and infant well-being combined with community impact and national leadership and advocacy for anti-racist approaches to improve the health and safety of Black mothers and children.
"We are delighted to have Dr. Crear-Perry join CSSP's Board of Directors," said Judith Meltzer, President of CSSP. "Dr. Crear-Perry's expertise as a physician and her work to increase equity in Black maternal and child health outcomes is strategically aligned with CSSP's mission to ensure racial, economic and social justice so that all children, families and communities can thrive."
Dr. Crear-Perry's work on maternal and infant health and access to affordable health care includes several positions; she has previously served as the Executive Director of the Birthing Project, Director of Women's and Children's Services at Jefferson Community Healthcare Center, and as the Director of Clinical Services for the City of New Orleans Health Department where she was responsible for four facilities that provided health care for the homeless, pediatric, WIC, and gynecologic services within the New Orleans clinical service area. Dr. Crear-Perry has been celebrated for her work to improve the availability and utilization of affordable health care for New Orleans' citizens post the Hurricane Katrina disaster of 2005.
The National Birth Equity Collaborative (NBEC) creates solutions that optimize Black maternal and infant health through training, policy advocacy, research, and community-centered collaboration with the goal that all Black mothers and babies thrive. Dr. Crear-Perry has long been active in working to reduce racial disparities in maternal health, including having appeared twice before the United Nations Office of the High Commissioner for Human Rights urging a human rights framework to improve maternal mortality.
Learn more about NBEC on their website. Read Dr. Crear-Perry's bio here.
About CSSP. The Center for the Study of Social Policy is a national, non-profit policy organization that connects community action, public system reform, and policy change to create a fair and just society. We work to achieve a racially, economically, and socially just society in which all children and families thrive by translating ideas into action, promoting public policies grounded in equity, supporting strong and inclusive communities, and advocating with and for all children and families marginalized by public policies and institutional practices.


text:
LeBron Scorches Nets for 49
Home / Game News / LeBron Scorches Nets for 49
By Heat Nation staff
(Brooklyn, NY) — In the words of Zach Lowe, "Sometimes the thing that matters most is that one team has the best player in the world, and the other team doesn't."
Immediately following Game 2, Paul Pierce went up to Head Coach Jason Kidd and said, "I want that assignment," referring to the defensive assignment on LeBron James. Pierce continued, "I think I've guarded him more than anybody in this gym. I know his tendencies a little bit more and I [want] to step up and lead the role that way."
Fast forward to Game 4, and it was evident that LBJ got the memo, as if to say, "So you think you can guard me one on one?" The King made sure to send a message with authority. Breakaway dunks, silky, agile drives through the key, tough fade aways, and bombs from deep that hit nothing but net, there was no stopping James on this particular eve. LeBron went off to the tune of 49 points on an incredible 16-24 shooting clip, while going to the line 19 times, showing how aggressive and in attack mode he was. When asked about how he was able to be so effective and make it look so easy tonight, James said, "I'm happy I was able to make enough plays for our team to win. It definitely wasn't easy. It may look that way, but it was hard work and dedication by all our guys."
LeBron actually had a chance to set a new playoff career high and end the game with 50, but split a pair at the line with 1.1 seconds remaining in the game. When asked about this, James responded with, "That's the first time that I've been disappointed in myself on a win…it's a free throw, and I hate missing free throws." Needless to say, Pierce finished the game with 16 points seven rebounds, five fouls, and a severely bruised ego.
The fourth quarter was a nail-baiter that never saw a team lead by more than three points, save for garbage free throws at the end of the game. After a Paul Pierce and-one to put the Brooklyn Nets up one with 4:25 remaining, LeBron nailed a go-ahead three pointer from about 28 feet out to put Miami up two. On the ensuing possession, James picked up his fourth foul and then a couple possessions later, he picked up his fifth. Joe Johnson, who finished the game with 18 points on a horrendous 5-15 shooting from the field (mostly because he was being guarded by James), said he wasn't "necessarily" trying to draw a sixth foul on LeBron. "I wasn't necessarily trying to draw a foul on him, but i thought he was being aggressive so i tried to use his aggressiveness against him."
This plan of attack was to no avail, as James made Johnson take a difficult fade away shot with a couple minutes to go. James said, "I take pride in my defense just as much as I do offensively. I believe in a one-on-one situation I can stop anyone from scoring."
With 1:15 remaining in the game, the Miami Heat were poised for a story book ending. Dwyane Wade, who finished the game with 15 points, four rebounds, and three assists, passed the ball to LeBron, who was quickly crowded in the painted area. After what seemed like a broken play out of a timeout, James found Mario Chalmers on the right wing, who quickly swung it to Chris Bosh in the right corner and from there it was all she wrote. Bosh nailed a critical three point shot to put the Heat up for good and although LeBron scored 49 points, Bosh's three was the most important of them all.
After the sequence had begun, James was asked about his thoughts when the ball started going in Bosh's direction. "Me setting the high pick and roll for D-Wade and they kind of messed up the coverage, so D-Wade hit me in the middle, I attacked KG and I knew what was about to happen. I spun in the paint and threw it to Rio, I knew it was gonna be good from CB." Bosh finished the game with 12 points, five rebounds, and three huge blocks.
The night before, James preached that his team needed to play better defense in order to win Game 4, and that's exactly what they did. After shooting 15-25 from three point land the night before, the Nets shot a mere 5-22 from the land of beyond, while shooting a total of 43.6% from the field as opposed to Miami who shot almost 53% from the field.
Game 5 is at home in what could be a close out game. LeBron stated the keys to the game perfectly. "Win. Play with a sense of urgency. Help each other and communicate. We do those things, we do it with a clear and aggressive mind, we give ourselves a good chance to win."
Brooklyn netscareer highgame 4lebron
← Previous Story Miami Heat-Brooklyn Nets Game Four Preview: Defining Game of Series
Next Story → Miami Heat-Brooklyn Nets Game Five Preview: Time to Wrap it Up
James Harden salutes Dwyane Wade for inspiring message of support after Nets debut
Here's the definitive reason why the Miami Heat weren't serious contenders to trade for James Harden
Jamal Crawford lists Dwyane Wade with Kobe Bryant and Allen Iverson as toughest players to guard
About Heat Nation staff
Heat Nation is your source for Miami Heat news, rumors, schedule, and videos for Heat fans everywhere.


text:
\section{Introduction}

Neural machine translation has recently shown impressive results \cite{kalchbrenner13emnlp,DBLP:conf/nips/SutskeverVL14,DBLP:journals/corr/BahdanauCB14}.
However, the translation of rare words is an open problem.
The vocabulary of neural models is typically limited to \num{30000}--\num{50000} words, but translation is an open-vocabulary problem,
and especially for languages with productive word formation processes such as agglutination and compounding, translation models require mechanisms that go below the word level.
As an example, consider compounds such as the German \emph{Abwasser\pipe behandlungs\pipe anlange} `sewage water treatment plant', for which a segmented, variable-length representation is intuitively more appealing than encoding the word as a fixed-length vector.

For word-level NMT models, the translation of out-of-vocabulary words has been addressed through a back-off to a dictionary look-up \cite{jean15,DBLP:journals/corr/LuongSLVZ14}.
We note that such techniques make assumptions that often do not hold true in practice.
For instance, there is not always a 1-to-1 correspondence between source and target words because of variance in the degree of morphological synthesis between languages, like in our introductory compounding example.
Also, word-level models are unable to translate or generate unseen words.
Copying unknown words into the target text, as done by \cite{jean15,DBLP:journals/corr/LuongSLVZ14}, is a reasonable strategy for names, but morphological changes and transliteration is often required, especially if alphabets differ.

We investigate NMT models that operate on the level of subword units.
Our main goal is to model open-vocabulary translation in the NMT network itself, without requiring a back-off model for rare words.
In addition to making the translation process simpler, we also find that the subword models achieve better accuracy for the translation of rare words than large-vocabulary models and back-off dictionaries,
and are able to productively generate new words that were not seen at training time.
Our analysis shows that the neural networks are able to learn compounding and transliteration from subword representations.

This paper has two main contributions:

\begin{itemize}
\setlength{\itemsep}{0pt}
\item We show that open-vocabulary neural machine translation is possible by encoding (rare) words via subword units.
We find our architecture simpler and more effective than using large vocabularies and back-off dictionaries \cite{jean15,DBLP:journals/corr/LuongSLVZ14}.

\item We adapt \emph{byte pair encoding} (BPE) \cite{Gage:1994:NAD:177910.177914}, a compression algorithm, to the task of word segmentation.
BPE allows for the representation of an open vocabulary through a fixed-size vocabulary of variable-length character sequences,
making it a very suitable word segmentation strategy for neural network models.

\end{itemize}

\section{Neural Machine Translation}

We follow the neural machine translation architecture by \newcite{DBLP:journals/corr/BahdanauCB14}, which we will briefly summarize here.
However, we note that our approach is not specific to this architecture.

The neural machine translation system is implemented as an encoder-decoder network with recurrent neural networks.

The encoder is a bidirectional neural network with gated recurrent units \cite{cho-EtAl:2014:EMNLP2014} that reads an input sequence $x=(x_1,...,x_m)$ and calculates a forward sequence of hidden states $(\overrightarrow{h}_1,...,\overrightarrow{h}_m)$,
and a backward sequence $(\overleftarrow{h}_1,...,\overleftarrow{h}_m)$.
The hidden states $\overrightarrow{h}_j$ and $\overleftarrow{h}_j$ are concatenated to obtain the annotation vector $h_j$.

The decoder is a recurrent neural network that predicts a target sequence $y=(y_1,...,y_n)$.
Each word $y_i$ is predicted based on a recurrent hidden state $s_i$, the previously predicted word $y_{i-1}$, and a context vector $c_i$.
$c_i$ is computed as a weighted sum of the annotations $h_j$.
The weight of each annotation $h_j$ is computed through an \emph{alignment model} $\alpha_{ij}$, which models the probability that $y_i$ is aligned to $x_j$.
The alignment model is a single-layer feedforward neural network that is learned jointly with the rest of the network through backpropagation.

A detailed description can be found in \cite{DBLP:journals/corr/BahdanauCB14}.
Training is performed on a parallel corpus with stochastic gradient descent.
For translation, a beam search with small beam size is employed.

\section{Subword Translation}

The main motivation behind this paper is that the translation of some words is transparent in that they are translatable by a competent translator even if they are novel to him or her, based on a translation of known subword units such as morphemes or phonemes.
Word categories whose translation is potentially transparent include:

\begin{itemize}
\setlength{\itemsep}{0pt}
\item named entities. Between languages that share an alphabet, names can often be copied from source to target text.
Transcription or transliteration may be required, especially if the alphabets or syllabaries differ. Example:\\
Barack Obama (English; German)\\
\foreignlanguage{russian}{Барак Обама} (Russian)\\
\begin{CJK*}{UTF8}{goth}
\small
\CJKnospace
\ignorespaces
バラク・オバマ
\end{CJK*}
(ba-ra-ku o-ba-ma) (Japanese)
\item cognates and loanwords. Cognates and loanwords with a common origin can differ in regular ways between languages, so that character-level translation rules are sufficient \cite{tiedemann:2012:EACL2012}. Example:\\
claustrophobia (English)\\
Klaustrophobie (German)\\
\foreignlanguage{russian}{Клаустрофобия} (Klaustrofobiâ) (Russian)
\item morphologically complex words. Words containing multiple morphemes, for instance formed via compounding, affixation, or inflection, may be translatable by translating the morphemes separately. Example:\\
solar system (English)\\
Sonnensystem (Sonne + System) (German)\\
Naprendszer (Nap + Rendszer) (Hungarian)
\end{itemize}

In an analysis of 100 rare tokens (not among the \num{50000} most frequent types) in our German training data\footnote{Primarily parliamentary proceedings and web crawl data.}, the majority of tokens are potentially translatable from English through smaller units.
We find 56 compounds, 21 names, 6 loanwords with a common origin (\emph{emancipate}$\to$\emph{emanzipieren}),
5 cases of transparent affixation (\emph{sweetish} `sweet' + `-ish' $\to$ \emph{süßlich} `süß' + `-lich'), 1 number and 1 computer language identifier.

Our hypothesis is that a segmentation of rare words into appropriate subword units is sufficient to allow for the neural translation network to learn transparent translations, and to generalize this knowledge to translate and produce unseen words.\footnote{Not every segmentation we produce is transparent. While we expect no performance benefit from opaque segmentations, i.e.\ segmentations where the units cannot be translated independently, our NMT models show robustness towards oversplitting.}
We provide empirical support for this hypothesis in Sections \ref{sec-eval} and \ref{sec-analysis}.
First, we discuss different subword representations.

\subsection{Related Work}
\label{related-sec}

For Statistical Machine Translation (SMT), the translation of unknown words has been the subject of intensive research.

A large proportion of unknown words are names, which can just be copied into the target text if both languages share an alphabet.
If alphabets differ, transliteration is required \cite{DBLP:conf/eacl/DurraniSHK14}.
Character-based translation has also been investigated with phrase-based models, which proved especially successful for closely related languages \cite{vilar-peter-ney:2007:WMT,Tiedemann474916,DBLP:conf/acl/NeubigWMK12}.

The segmentation of morphologically complex words such as compounds is widely used for SMT, and various algorithms for morpheme segmentation have been investigated \cite{Niessen00improvingsmt,koehn03b,virpioja07mtsummit,DBLP:conf/acl/StallardDKLB12}.
Segmentation algorithms commonly used for phrase-based SMT tend to be conservative in their splitting decisions,
whereas we aim for an aggressive segmentation that allows for open-vocabulary translation with a compact network vocabulary, and without having to resort to back-off dictionaries.

The best choice of subword units may be task-specific.
For speech recognition, phone-level language models have been used \cite{DBLP:conf/interspeech/BazziG00}.
\newcite{mikolov2012} investigate subword language models, and propose to use syllables.
For multilingual segmentation tasks, multilingual algorithms have been proposed \cite{snyder-barzilay:2008:ACLMain}.
We find these intriguing, but inapplicable at test time.

Various techniques have been proposed to produce fixed-length continuous word vectors based on characters or morphemes \cite{DBLP:conf/conll/LuongSM13,Botha2014,ling15,DBLP:journals/corr/KimJSR15}.
An effort to apply such techniques to NMT, parallel to ours, has found no significant improvement over word-based approaches \cite{2015arXiv151104586L}.
One technical difference from our work is that the attention mechanism still operates on the level of words in the model by \newcite{2015arXiv151104586L}, and that the representation of each word is fixed-length.
We expect that the attention mechanism benefits from our variable-length representation: the network can learn to place attention on different subword units at each step.
Recall our introductory example \emph{Abwasserbehandlungsanlange}, for which a subword segmentation avoids the information bottleneck of a fixed-length representation.

Neural machine translation differs from phrase-based methods in that there are strong incentives to minimize the vocabulary size of neural models to increase time and space efficiency, and to allow for translation without back-off models.
At the same time, we also want a compact representation of the text itself, since an increase in text length reduces efficiency and increases the distances over which neural models need to pass information.

A simple method to manipulate the trade-off between vocabulary size and text size is to use shortlists of unsegmented words, using subword units only for rare words.
As an alternative, we propose a segmentation algorithm based on byte pair encoding (BPE), which lets us learn a vocabulary that provides a good compression rate of the text.

\subsection{Byte Pair Encoding (BPE)}

Byte Pair Encoding (BPE) \cite{Gage:1994:NAD:177910.177914} is a simple data compression technique that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte.
We adapt this algorithm for word segmentation.
Instead of merging frequent pairs of bytes, we merge characters or character sequences.

Firstly, we initialize the symbol vocabulary with the character vocabulary, and represent each word as a sequence of characters, plus a special end-of-word symbol `$\cdot$', which allows us to restore the original tokenization after translation.
We iteratively count all symbol pairs and replace each occurrence of the most frequent pair (`A', `B') with a new symbol `AB'.
Each merge operation produces a new symbol which represents a character $n$-gram.
Frequent character $n$-grams (or whole words) are eventually merged into a single symbol, thus BPE requires no shortlist.
The final symbol vocabulary size is equal to the size of the initial vocabulary, plus the number of merge operations -- the latter is the only hyperparameter of the algorithm.

\begin{algorithm}
\begin{lstlisting}
import re, collections

def get_stats(vocab):
  pairs = collections.defaultdict(int)
  for word, freq in vocab.items():
    symbols = word.split()
    for i in range(len(symbols)-1):
      pairs[symbols[i],symbols[i+1]] += freq
  return pairs

def merge_vocab(pair, v_in):
  v_out = {}
  bigram = re.escape(' '.join(pair))
  p = re.compile(r'(?<!\S)' + bigram + r'(?!\S)')
  for word in v_in:
    w_out = p.sub(''.join(pair), word)
    v_out[w_out] = v_in[word]
  return v_out

vocab = {'l o w </w>' : 5, 'l o w e r </w>' : 2,
         'n e w e s t </w>':6, 'w i d e s t </w>':3}
num_merges = 10
for i in range(num_merges):
  pairs = get_stats(vocab)
  best = max(pairs, key=pairs.get)
  vocab = merge_vocab(best, vocab)
  print(best)
\end{lstlisting}
\caption{Learn BPE operations}
\label{bpe-algorithm}
\end{algorithm}

For efficiency, we do not consider pairs that cross word boundaries.
The algorithm can thus be run on the dictionary extracted from a text, with each word being weighted by its frequency.
A minimal Python implementation is shown in Algorithm \ref{bpe-algorithm}.
In practice, we increase efficiency by indexing all pairs, and updating data structures incrementally.

The main difference to other compression algorithms, such as Huffman encoding, which have been proposed to produce a variable-length encoding of words for NMT \cite{chitnis15},
is that our symbol sequences are still interpretable as subword units, and that the network can generalize to translate and produce new words (unseen at training time) on the basis of these subword units.

\begin{figure}
\centering
\small{
\begin{tabular}{lcl}
r $\cdot$ & $\rightarrow$ & r$\cdot$ \\
l o & $\rightarrow$ & lo\\
lo w & $\rightarrow$ & low\\
e r$\cdot$ & $\rightarrow$ & er$\cdot$ \\
\end{tabular}}
\caption{BPE merge operations learned from dictionary \{`low', `lowest', `newer', `wider'\}.}
\label{bpe-example}
\end{figure}

Figure \ref{bpe-example} shows a toy example of learned BPE operations.
At test time, we first split words into sequences of characters, then apply the learned operations to merge the characters into larger, known symbols.
This is applicable to any word, and allows for open-vocabulary networks with fixed symbol vocabularies.\footnote{The only symbols that will be unknown at test time are unknown characters, or symbols of which all occurrences in the training text have been merged into larger symbols, like `safeguar', which has all occurrences in our training text merged into `safeguard'. We observed no such symbols at test time, but the issue could be easily solved by recursively reversing specific merges until all symbols are known.}
In our example, the OOV `lower' would be segmented into `low er$\cdot$'.

We evaluate two methods of applying BPE:
learning two independent encodings, one for the source, one for the target vocabulary, or learning the encoding on the union of the two vocabularies (which we call \emph{joint BPE}).\footnote{In practice, we simply concatenate the source and target side of the training set to learn joint BPE.}
The former has the advantage of being more compact in terms of text and vocabulary size, and having stronger guarantees that each subword unit has been seen in the training text of the respective language,
whereas the latter improves consistency between the source and the target segmentation.
If we apply BPE independently, the same name may be segmented differently in the two languages, which makes it harder for the neural models to learn a mapping between the subword units.
To increase the consistency between English and Russian segmentation despite the differing alphabets, we transliterate the Russian vocabulary into Latin characters with ISO-9 to learn the joint BPE encoding,
then transliterate the BPE merge operations back into Cyrillic to apply them to the Russian training text.\footnote{Since the Russian training text also contains words that use the Latin alphabet, we also apply the Latin BPE operations.}

\section{Evaluation}
\label{sec-eval}

We aim to answer the following empirical questions:

\begin{itemize}
\setlength{\itemsep}{0pt}
\item Can we improve the translation of rare and unseen words in neural machine translation by representing them via subword units?
\item Which segmentation into subword units performs best in terms of vocabulary size, text size, and translation quality?
\end{itemize}

We perform experiments on data from the shared translation task of WMT 2015.
For English$\to$German, our training set consists of 4.2 million sentence pairs, or approximately 100 million tokens.
For English$\to$Russian, the training set consists of 2.6 million sentence pairs, or approximately 50 million tokens.
We tokenize and truecase the data with the scripts provided in Moses \cite{koehnmoses}.
We use newstest2013 as development set, and report results on newstest2014 and newstest2015.

We report results with {\sc Bleu} (\emph{mteval-v13a.pl}),
and {\sc chrF3} \cite{popovic:2015:WMT}, a character n-gram F$_3$ score which was found to correlate well with human judgments, especially for translations out of English \cite{stanojevic-EtAl:2015:WMT}.
Since our main claim is concerned with the translation of rare and unseen words, we report separate statistics for these.
We measure these through unigram F$_1$, which we calculate as the harmonic mean of clipped unigram precision and recall.\footnote{Clipped unigram precision is essentially 1-gram BLEU without brevity penalty.}

We perform all experiments with Groundhog\footnote{\url{github.com/sebastien-j/LV_groundhog}} \cite{DBLP:journals/corr/BahdanauCB14}.
We generally follow settings by previous work \cite{DBLP:journals/corr/BahdanauCB14,jean15}.
All networks have a hidden layer size of 1000, and an embedding layer size of 620.
Following \newcite{jean15}, we only keep a shortlist of $\tau=30000$ words in memory.

During training, we use Adadelta \cite{DBLP:journals/corr/abs-1212-5701}, a minibatch size of 80, and reshuffle the training set between epochs.
We train a network for approximately 7 days, then take the last 4 saved models (models being saved every 12 hours), and continue training each with a fixed embedding layer (as suggested by \cite{jean15}) for 12 hours.
We perform two independent training runs for each models, once with cut-off for gradient clipping \cite{DBLP:conf/icml/PascanuMB13} of 5.0, once with a cut-off of 1.0 -- the latter produced better single models for most settings.
We report results of the system that performed best on our development set (newstest2013), and of an ensemble of all 8 models.

We use a beam size of 12 for beam search, with probabilities normalized by sentence length.
We use a bilingual dictionary based on fast-align \cite{dyer-chahuneau-smith:2013:NAACL-HLT}.
For our baseline, this serves as back-off dictionary for rare words.
We also use the dictionary to speed up translation for all experiments, only performing the softmax over a filtered list of candidate translations (like \newcite{jean15}, we use $K=30000$; $K'=10$).

\subsection{Subword statistics}

Apart from translation quality, which we will verify empirically, our main objective is to represent an open vocabulary through a compact fixed-size subword vocabulary, and allow for efficient training and decoding.\footnote{The time complexity of encoder-decoder architectures is at least linear to sequence length, and oversplitting harms efficiency.}

Statistics for different segmentations of the German side of the parallel data are shown in Table \ref{splitting}.
A simple baseline is the segmentation of words into character $n$-grams.\footnote{Our character n-grams do not cross word boundaries. We mark whether a subword is word-final or not with a special character, which allows us to restore the original tokenization.}
Character $n$-grams allow for different trade-offs between sequence length (\# tokens) and vocabulary size (\# types), depending on the choice of $n$.
The increase in sequence length is substantial; one way to reduce sequence length is to leave a shortlist of the $k$ most frequent word types unsegmented.
Only the unigram representation is truly open-vocabulary.
However, the unigram representation performed poorly in preliminary experiments, and we report translation results with a bigram representation, which is empirically better, but unable to produce some tokens in the test set with the training set vocabulary.

We report statistics for several word segmentation techniques that have proven useful in previous SMT research, including
frequency-based compound splitting \cite{koehn03b}, rule-based hyphenation \cite{Liang:151530}, and Morfessor \cite{creutz-lagus:2002:ACL02-MPL}.
We find that they only moderately reduce vocabulary size, and do not solve the unknown word problem, and we thus find them unsuitable for our goal of open-vocabulary translation without back-off dictionary.

BPE meets our goal of being open-vocabulary, and the learned merge operations can be applied to the test set to obtain a segmentation with no unknown symbols.\footnote{Joint BPE can produce segments that are unknown because they only occur in the English training text, but these are rare (0.05\% of test tokens).}
Its main difference from the character-level model is that the more compact representation of BPE allows for shorter sequences, and that the attention model operates on variable-length units.\footnote{We highlighted the limitations of word-level attention in section \ref{related-sec}. At the other end of the spectrum, the character level is suboptimal for alignment \cite{Tiedemann474916}.}
Table \ref{splitting} shows BPE with \num{59500} merge operations, and joint BPE with \num{89500} operations.

In practice, we did not include infrequent subword units in the NMT network vocabulary, since there is noise in the subword symbol sets, e.g.\ because of characters from foreign alphabets.
Hence, our network vocabularies in Table \ref{results} are typically slightly smaller than the number of types in Table \ref{splitting}.

\begin{table}
\centering
\small{
\begin{tabular}{lrrrr}
segmentation & \# tokens & \# types &\# UNK \\
\hline
none & 100 m & \num{1750000} & 1079 \\
characters & 550 m & \num{3000} & 0\\
character bigrams & 306 m & \num{20000} & 34\\
character trigrams & 214 m & \num{120000} & 59 \\
\hline
compound splitting$^\triangle$ & 102 m & \num{1100000} & 643 \\
morfessor* & 109 m & \num{544000} & 237\\
hyphenation$^\diamond$ & 186 m & \num{404000} & 230\\
\hline
BPE & 112 m & \num{63000} & 0 \\
BPE (joint) & 111 m & \num{82000} & 32 \\
\hline
character bigrams & \multirow{2}{*}{129 m} & \multirow{2}{*}{\num{69000}} & \multirow{2}{*}{34} \\
(shortlist: \num{50000}) \\
\end{tabular}}
\caption{Corpus statistics for German training corpus with different word segmentation techniques. \#UNK: number of unknown tokens in newstest2013. $\triangle$: \cite{koehn03b}; *: \cite{creutz-lagus:2002:ACL02-MPL}; $\diamond$: \cite{Liang:151530}.}
\label{splitting}
\end{table}

\subsection{Translation experiments}

\begin{table*}
\centering
\small
\setlength{\tabcolsep}{3pt}
\begin{tabular}{llrrr|rr|rr|rrr}
&&&\multicolumn{2}{c|}{vocabulary}&\multicolumn{2}{c|}{{\sc Bleu}} & \multicolumn{2}{c|}{{\sc chrF3}} & \multicolumn{3}{c}{unigram F$_1$ (\%)}\\
name & segmentation & shortlist & source & target & single & ens-8 & single & ens-8 & all & rare & OOV\\
\hline
\multicolumn{4}{l}{syntax-based \cite{sennrichhaddow15}} & &
24.4 & - &
55.3 & - & 59.1 & 46.0 & 37.7\\
\hline
WUnk & - & - & \num{300000} & \num{500000} &
20.6 & 22.8 &
47.2 & 48.9 & 56.7 & 20.4 & 0.0\\
WDict & - & - & \num{300000} & \num{500000} &
22.0 & 24.2 &
50.5 & 52.4 & 58.1 & 36.8 & \textbf{36.8}\\
C2-50k &char-bigram & \num{50000} & \num{60000} & \num{60000} &
\textbf{22.8} & \textbf{25.3} &
51.9 & 53.5 & 58.4 & 40.5 & 30.9 \\
BPE-60k &BPE & - & \num{60000} & \num{60000} &
21.5 & 24.5 &
\textbf{52.0} & 53.9 & 58.4 & 40.9 & 29.3\\
BPE-J90k &BPE (joint) & - & \num{90000} & \num{90000} &
\textbf{22.8} & 24.7 &
51.7 & \textbf{54.1} & \textbf{58.5} & \textbf{41.8} & 33.6\\
\end{tabular}
\caption{English$\to$German translation performance ({\sc Bleu}, {\sc chrF3} and unigram F$_1$) on newstest2015.
Ens-8: ensemble of 8 models.
Best NMT system in bold.
Unigram F$_1$ (with ensembles) is computed for all words ($n=44085$), rare words (not among top \num{50000} in training set; $n=2900$), and OOVs (not in training set; $n=1168$).}
\label{results}
\end{table*}

English$\to$German translation results are shown in Table \ref{results};
English$\to$Russian results in Table \ref{results-russian}.

Our baseline \textbf{WDict} is a word-level model with a back-off dictionary.
It differs from \textbf{WUnk} in that the latter uses no back-off dictionary, and just represents out-of-vocabulary words as UNK\footnote{We use \emph{UNK} for words that are outside the model vocabulary, and \emph{OOV} for those that do not occur in the training text.}.
The back-off dictionary improves unigram F$_1$ for rare and unseen words, although the improvement is smaller for English$\to$Russian, since the back-off dictionary is incapable of transliterating names.

All subword systems operate without a back-off dictionary.
We first focus on unigram F$_1$, where all systems improve over the baseline,
especially for rare words (36.8\%$\rightarrow$41.8\% for EN$\to$DE; 26.5\%$\rightarrow$29.7\% for EN$\to$RU).
For OOVs, the baseline strategy of copying unknown words works well for English$\to$German.
However, when alphabets differ, like in English$\to$Russian, the subword models do much better.

Unigram F$_1$ scores indicate that learning the BPE symbols on the vocabulary union (\textbf{BPE-J90k})
is more effective than learning them separately (\textbf{BPE-60k}), and more effective than using character bigrams with a shortlist of \num{50000} unsegmented words (\textbf{C2-50k}),
but all reported subword segmentations are viable choices and outperform the back-off dictionary baseline.

Our subword representations cause big improvements in the translation of rare and unseen words, but these only constitute 9-11\% of the test sets.
Since rare words tend to carry central information in a sentence, we suspect that {\sc Bleu} and {\sc chrF3} underestimate their effect on translation quality.
Still, we also see improvements over the baseline in total unigram F$_1$, as well as {\sc Bleu} and {\sc chrF3}, and the subword ensembles outperform the WDict baseline by 0.3--1.3 {\sc Bleu} and 0.6--2 {\sc chrF3}.
There is some inconsistency between {\sc Bleu} and {\sc chrF3}, which we attribute to the fact that {\sc Bleu} has a precision bias, and {\sc chrF3} a recall bias.

For English$\to$German, we observe the best {\sc Bleu} score of 25.3 with C2-50k, but the best {\sc chrF3} score of 54.1 with BPE-J90k.
For comparison to the (to our knowledge) best non-neural MT system on this data set, we report syntax-based SMT results \cite{sennrichhaddow15}.
We observe that our best systems outperform the syntax-based system in terms of {\sc Bleu}, but not in terms of {\sc chrF3}.
Regarding other neural systems, \newcite{luong-pham-manning:2015:EMNLP} report a {\sc Bleu} score of 25.9 on newstest2015, but we note that they use an ensemble of 8 independently trained models, and also report strong improvements from applying dropout, which we did not use.
We are confident that our improvements to the translation of rare words are orthogonal to improvements achievable through other improvements in the network architecture, training algorithm, or better ensembles.

For English$\to$Russian, the state of the art is the phrase-based system by \newcite{haddow-EtAl:2015:WMT}.
It outperforms our WDict baseline by 1.5 {\sc Bleu}.
The subword models are a step towards closing this gap, and BPE-J90k yields an improvement of 1.3 {\sc Bleu}, and 2.0 {\sc chrF3}, over WDict.

\begin{table*}
\centering
\small
\setlength{\tabcolsep}{3pt}
\begin{tabular}{llrrr|rr|rr|rrr}
&&&\multicolumn{2}{c|}{vocabulary}&\multicolumn{2}{c|}{{\sc Bleu}} & \multicolumn{2}{c|}{{\sc chrF3}} & \multicolumn{3}{c}{unigram F$_1$ (\%)}\\
name & segmentation & shortlist & source & target & single & ens-8 & single & ens-8 & all & rare & OOV\\
\hline
\multicolumn{3}{l}{phrase-based \cite{haddow-EtAl:2015:WMT}} & & &
24.3 & - &
53.8 & - & 56.0 & 31.3 & 16.5\\
\hline
WUnk & - & - & \num{300000} & \num{500000} &
18.8 & 22.4 &
46.5 & 49.9 & 54.2 & 25.2 & 0.0\\
WDict & - & - & \num{300000} & \num{500000} &
19.1 & 22.8 &
47.5 & 51.0 & 54.8 & 26.5 & 6.6\\
C2-50k & char-bigram & \num{50000} & \num{60000} & \num{60000} &
\textbf{20.9} & \textbf{24.1} &
49.0 & 51.6 & 55.2 & 27.8 & 17.4\\
BPE-60k & BPE & - & \num{60000} & \num{60000} &
20.5 & 23.6 &
\textbf{49.8} & 52.7 & 55.3 & 29.7 & 15.6 \\
BPE-J90k & BPE (joint)& - & \num{90000} & \num{100000} &
20.4 & \textbf{24.1} &
49.7 & \textbf{53.0} & \textbf{55.8} & \textbf{29.7} & \textbf{18.3}\\
\end{tabular}
\caption{English$\to$Russian translation performance ({\sc Bleu}, {\sc chrF3} and unigram F$_1$) on newstest2015.
Ens-8: ensemble of 8 models.
Best NMT system in bold.
Unigram F$_1$ (with ensembles) is computed for all words ($n=55654$), rare words (not among top \num{50000} in training set; $n=5442$), and OOVs (not in training set; $n=851$).}
\label{results-russian}
\end{table*}

As a further comment on our translation results, we want to emphasize that performance variability is still an open problem with NMT.
On our development set, we observe differences of up to 1 {\sc Bleu} between different models.
For single systems, we report the results of the model that performs best on dev (out of 8), which has a stabilizing effect, but how to control for randomness deserves further attention in future research.

\section{Analysis}
\label{sec-analysis}

\subsection{Unigram accuracy}

Our main claims are that the translation of rare and unknown words is poor in word-level NMT models, and that subword models improve the translation of these word types.
To further illustrate the effect of different subword segmentations on the translation of rare and unseen words, we plot target-side words sorted by their frequency in the training set.\footnote{We perform binning of words with the same training set frequency, and apply bezier smoothing to the graph.}
To analyze the effect of vocabulary size, we also include the system \textbf{C2-3/500k}, which is a system with the same vocabulary size as the WDict baseline, and character bigrams to represent unseen words.

Figure \ref{f1-figure} shows results for the English--German ensemble systems on newstest2015.
Unigram F$_1$ of all systems tends to decrease for lower-frequency words.
The baseline system has a spike in F$_1$ for OOVs, i.e.\ words that do not occur in the training text.
This is because a high proportion of OOVs are names, for which a copy from the source to the target text is a good strategy for English$\to$German.

The systems with a target vocabulary of \num{500000} words mostly differ in how well they translate words with rank > \num{500000}.
A back-off dictionary is an obvious improvement over producing UNK, but the subword system C2-3/500k achieves better performance.
Note that all OOVs that the back-off dictionary produces are words that are copied from the source, usually names, while the subword systems can productively form new words such as compounds.

For the \num{50000} most frequent words, the representation is the same for all neural networks, and all neural networks achieve comparable unigram F$_1$ for this category.
For the interval between frequency rank \num{50000} and \num{500000}, the comparison between C2-3/500k and C2-50k unveils an interesting difference.
The two systems only differ in the size of the shortlist, with C2-3/500k representing words in this interval as single units, and C2-50k via subword units.
We find that the performance of C2-3/500k degrades heavily up to frequency rank \num{500000}, at which point the model switches to a subword representation and performance recovers.
The performance of C2-50k remains more stable.
We attribute this to the fact that subword units are less sparse than words.
In our training set, the frequency rank \num{50000} corresponds to a frequency of 60 in the training data; the frequency rank \num{500000} to a frequency of 2.
Because subword representations are less sparse, reducing the size of the network vocabulary, and representing more words via subword units, can lead to better performance.

The F$_1$ numbers hide some qualitative differences between systems.
For English$\to$German, WDict produces few OOVs (26.5\% recall), but with high precision (60.6\%) , whereas the subword systems achieve higher recall, but lower precision.
We note that the character bigram model C2-50k produces the most OOV words, and achieves relatively low precision of 29.1\% for this category.
However, it outperforms the back-off dictionary in recall (33.0\%).
BPE-60k, which suffers from transliteration (or copy) errors due to segmentation inconsistencies, obtains a slightly better precision (32.4\%), but a worse recall (26.6\%).
In contrast to BPE-60k, the joint BPE encoding of BPE-J90k improves both precision (38.6\%) and recall (29.8\%).

For English$\to$Russian, unknown names can only rarely be copied, and usually require transliteration.
Consequently, the WDict baseline performs more poorly for OOVs (9.2\% precision; 5.2\% recall), and the subword models improve both precision and recall (21.9\% precision and 15.6\% recall for BPE-J90k).
The full unigram F$_1$ plot is shown in Figure \ref{f1-figure2}.

\begin{figure}

\begin{tikzpicture}[scale=0.9]
\pgfplotsset{major grid style={style=dotted,color=black!20}}
\begin{semilogxaxis}[xlabel=training set frequency rank,
    xmin=1,
    xmax=2200000,
    ymin=0,
    ymax=1,
    ylabel=unigram F$_1$,
    legend pos = south west,
    legend style={
        font=\scriptsize,
        /tikz/nodes={anchor=west}
        },
    mark size = 0.1,
    ]

    \addplot +[orange, no markers, raw gnuplot, solid, line width=0.2ex, id=de-bpej] gnuplot {set samples 2000,2000; plot 'data/german-33+36.plot' smooth sbezier;};
    \addplot +[red, no markers, raw gnuplot, dotted, line cap=round, line width=0.2ex, id=de-50k] gnuplot {set samples 2000,2000; plot 'data/german-16+39.plot' smooth sbezier;};
    \addplot +[blue, no markers, raw gnuplot, dashed, line width=0.2ex, id=de-500k] gnuplot {set samples 2000,2000; plot 'data/german-14+38.plot' smooth sbezier;};
    \addplot +[black, no markers, raw gnuplot, line width=0.15ex, id=de-500kdict] gnuplot {set samples 2000,2000; plot 'data/german-mapped.19+35.plot' smooth sbezier;};
    \addplot +[black, no markers, raw gnuplot, dotted, line width=0.2ex, id=de-500knone] gnuplot {set samples 2000,2000; plot 'data/german-19+35.plot' smooth sbezier;};

    \addlegendentry{BPE-J90k}
    \addlegendentry{C2-50k}
    \addlegendentry{C2-300/500k}
    \addlegendentry{WDict}
    \addlegendentry{WUnk}

    \draw[black!20] (axis cs:50000,0) -- (axis cs:50000,0.9) node [gray, above] {\scriptsize{\num{50000}}};
    \draw[black!20] (axis cs:500000,0) -- (axis cs:500000,0.9) node [gray, above] {\scriptsize{\num{500000}}};

\end{semilogxaxis}
\end{tikzpicture}
\caption{English$\to$German unigram F$_1$ on newstest2015 plotted by training set frequency rank for different NMT systems.}
\label{f1-figure}
\end{figure}

\begin{figure}

\begin{tikzpicture}[scale=0.9]
\pgfplotsset{major grid style={style=dotted,color=black!20}}
\begin{semilogxaxis}[xlabel=training set frequency rank,
    xmin=1,
    xmax=2000000,
    ymin=0,
    ymax=1,
    ylabel=unigram F$_1$,
    legend pos = south west,
    legend style={
        font=\scriptsize,
        /tikz/nodes={anchor=west}
        },
    mark size = 0.1,
    ]

    \addplot +[orange, no markers, raw gnuplot, solid, line width=0.2ex, id=ru-bpej] gnuplot {set samples 2000,2000; plot 'data/russian-6+8.plot' smooth sbezier;};
    \addplot +[red, no markers, raw gnuplot, dotted, line cap=round, line width=0.2ex, id=ru-50k] gnuplot {set samples 2000,2000; plot 'data/russian-3+10.plot' smooth sbezier;};
    \addplot +[black, no markers, raw gnuplot, line width=0.15ex, id=ru-500kdict] gnuplot {set samples 2000,2000; plot 'data/russian-mapped.1+9.plot' smooth sbezier;};
    \addplot +[black, no markers, raw gnuplot, dotted, line width=0.2ex, id=ru-500knone] gnuplot {set samples 2000,2000; plot 'data/russian-1+9.plot' smooth sbezier;};

    \addlegendentry{BPE-J90k}
    \addlegendentry{C2-50k}
    \addlegendentry{WDict}
    \addlegendentry{WUnk}

    \draw[black!20] (axis cs:50000,0) -- (axis cs:50000,0.9) node [gray, above] {\scriptsize{\num{50000}}};
    \draw[black!20] (axis cs:500000,0) -- (axis cs:500000,0.9) node [gray, above] {\scriptsize{\num{500000}}};

\end{semilogxaxis}
\end{tikzpicture}
\caption{English$\to$Russian unigram F$_1$ on newstest2015 plotted by training set frequency rank for different NMT systems.}
\label{f1-figure2}
\end{figure}

\subsection{Manual Analysis}

Table \ref{examples-de} shows two translation examples for the translation direction English$\to$German,
Table \ref{examples-ru} for English$\to$Russian.
The baseline system fails for all of the examples, either by deleting content (\emph{health}), or by copying source words that should be translated or transliterated.
The subword translations of \emph{health research institutes} show that the subword systems are capable of learning translations when oversplitting (\emph{research}$\rightarrow$\emph{Fo\pipe rs\pipe ch\pipe un\pipe g}),
or when the segmentation does not match morpheme boundaries: the segmentation \emph{Forschungs\pipe instituten} would be linguistically more plausible, and simpler to align to the English \emph{research institutes},
than the segmentation \emph{Forsch\pipe ungsinstitu\pipe ten} in the BPE-60k system, but still, a correct translation is produced.
If the systems have failed to learn a translation due to data sparseness, like for \emph{asinine}, which should be translated as \emph{dumm}, we see translations that are wrong, but could be plausible for (partial) loanwords (\emph{asinine Situation}$\rightarrow$\emph{Asinin-Situation}).

The English$\to$Russian examples show that the subword systems are capable of transliteration.
However, transliteration errors do occur, either due to ambiguous transliterations, or because of non-consistent segmentations between source and target text which make it hard for the system to learn a transliteration mapping.
Note that the BPE-60k system encodes \emph{Mirzayeva} inconsistently for the two language pairs (\emph{Mirz\pipe ayeva}$\rightarrow$\foreignlanguage{russian}{Мир\pipe за\pipe ева} \emph{Mir\pipe za\pipe eva}).
This example is still translated correctly, but we observe spurious insertions and deletions of characters in the BPE-60k system.
An example is the transliteration of \emph{rakfisk}, where a \emph{\foreignlanguage{russian}{п}} is inserted and a \emph{\foreignlanguage{russian}{к}} is deleted.
We trace this error back to translation pairs in the training data with inconsistent segmentations, such as (\emph{p\pipe rak\pipe ri\pipe ti}$\rightarrow$\emph{\foreignlanguage{russian}{пра\pipe крит\pipe и}} (pra\pipe krit\pipe i)), from which the translation (\emph{rak}$\rightarrow$\emph{\foreignlanguage{russian}{пра}}) is erroneously learned.
The segmentation of the joint BPE system (BPE-J90k) is more consistent (\emph{pra\pipe krit\pipe i}$\rightarrow$\emph{\foreignlanguage{russian}{пра\pipe крит\pipe и}} (pra\pipe krit\pipe i)).

\begin{table}
\centering
\small
\begin{tabular}{l|l}
system & sentence\\
\hline
source & health research institutes \\
reference & Gesundheitsforschungsinstitute \\
WDict & Forschungsinstitute \\
C2-50k & Fo\pipe rs\pipe ch\pipe un\pipe gs\pipe in\pipe st\pipe it\pipe ut\pipe io\pipe ne\pipe n\\
BPE-60k & Gesundheits\pipe forsch\pipe ungsinstitu\pipe ten \\
BPE-J90k & Gesundheits\pipe forsch\pipe ungsin\pipe stitute \\
\hline
source & asinine situation \\
reference & dumme Situation \\
WDict & asinine situation $\rightarrow$ UNK $\rightarrow$ asinine \\
C2-50k & as\pipe in\pipe in\pipe e situation $\rightarrow$ As\pipe in\pipe en\pipe si\pipe tu\pipe at\pipe io\pipe n\\
BPE-60k & as\pipe in\pipe ine situation $\rightarrow$ A\pipe in\pipe line-\pipe Situation \\
BPE-J90K & as\pipe in\pipe ine situation $\rightarrow$ As\pipe in\pipe in-\pipe Situation \\
\end{tabular}
\caption{English$\to$German translation example. ``\pipe '' marks subword boundaries.}
\label{examples-de}
\end{table}

\begin{table}
\centering
\small
\begin{tabular}{l|l}
system & sentence\\
\hline
source & Mirzayeva \\
reference & \foreignlanguage{russian}{Мирзаева} (Mirzaeva)\\
WDict & Mirzayeva\phantom{\pipe\pipe\pipe\pipe} $\rightarrow$ UNK $\rightarrow$ Mirzayeva \\
C2-50k & Mi\pipe rz\pipe ay\pipe ev\pipe a $\rightarrow$ \foreignlanguage{russian}{Ми\pipe рз\pipe ае\pipe ва} (Mi\pipe rz\pipe ae\pipe va)\\
BPE-60k & Mirz\pipe ayeva\phantom{\pipe\pipe\pipe} $\rightarrow$ \foreignlanguage{russian}{Мир\pipe за\pipe ева} (Mir\pipe za\pipe eva)\\
BPE-J90k & Mir\pipe za\pipe yeva\phantom{\pipe\pipe} $\rightarrow$ \foreignlanguage{russian}{Мир\pipe за\pipe ева} (Mir\pipe za\pipe eva)\\
\hline
source & rakfisk \\
reference & \foreignlanguage{russian}{ракфиска} (rakfiska)\\
WDict & rakfisk\phantom{\pipe\pipe\pipe} $\rightarrow$ UNK $\rightarrow$ rakfisk\\
C2-50k & ra\pipe kf\pipe is\pipe k $\rightarrow$ \foreignlanguage{russian}{ра\pipe кф\pipe ис\pipe к} (ra\pipe kf\pipe is\pipe k)\\
BPE-60k & rak\pipe f\pipe isk\phantom{\pipe} $\rightarrow$ \foreignlanguage{russian}{пра\pipe ф\pipe иск} (pra\pipe f\pipe isk)\\
BPE-J90k & rak\pipe f\pipe isk\phantom{\pipe} $\rightarrow$ \foreignlanguage{russian}{рак\pipe ф\pipe иска} (rak\pipe f\pipe iska) \\
\end{tabular}
\caption{English$\to$Russian translation examples. ``\pipe '' marks subword boundaries.}
\label{examples-ru}
\end{table}

\section{Conclusion}

The main contribution of this paper is that we show that neural machine translation systems are capable of open-vocabulary translation by representing rare and unseen words as a sequence of subword units.\footnote{The source code of the segmentation algorithms is available at \url{https://github.com/rsennrich/subword-nmt}.}
This is both simpler and more effective than using a back-off translation model.
We introduce a variant of byte pair encoding for word segmentation, which is capable of encoding open vocabularies with a compact symbol vocabulary of variable-length subword units.
We show performance gains over the baseline with both BPE segmentation, and a simple character bigram segmentation.

Our analysis shows that not only out-of-vocabulary words, but also rare in-vocabulary words are translated poorly by our baseline NMT system,
and that reducing the vocabulary size of subword models can actually improve performance.
In this work, our choice of vocabulary size is somewhat arbitrary, and mainly motivated by comparison to prior work.
One avenue of future research is to learn the optimal vocabulary size for a translation task, which we expect to depend on the language pair and amount of training data, automatically.
We also believe there is further potential in bilingually informed segmentation algorithms to create more alignable subword units, although the segmentation algorithm cannot rely on the target text at runtime.

While the relative effectiveness will depend on language-specific factors such as vocabulary size, we believe that subword segmentations are suitable for most language pairs, eliminating the need for large NMT vocabularies or back-off models.

\section*{Acknowledgments}

We thank Maja Popović for her implementation of {\sc chrF}, with which we verified our re-implementation.
The research presented in this publication was conducted in cooperation with Samsung Electronics Polska sp.\ z o.o.\ - Samsung R\&D Institute Poland.
This project received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement 645452 (QT21).

\bibliographystyle{acl2016}

redpajama_set_name: RedPajamaArXiv