Q:

Apostrophes around words are the wrong way round

If I use apostrophes around a word, the first one is the wrong way round. For example, 
'word'

appears as 
’word’

A:

Summarising the comments, your input is incorrect it should be
`word'

or, with an up-to-date TeX distribution,
‘word’

using the Unicode quotes U+2018 LEFT SINGLE QUOTATION MARK and U+2019 RIGHT SINGLE QUOTATION MARK.
However, it is often easier to use the textcmds package and write
\q{word}

or the more sophisticated csquotes package, and so write
\enquote{word} 

or 
\enquote*{word}

What about math? Well, with unicode-math (with XeLaTeX or LuaLaTeX) an input such as
\documentclass{article}
\usepackage{unicode-math}

\begin{document}

$\mathrm{‘E’}$

\end{document}

yields

With pdflatex it's a bit more complicated:
\documentclass{article}

\newcommand{\vmathrm}[1]{
  \mathrm{%
    \def\textquoteleft{`}%
    \def\textquoteright{'}%
    \mathcode`\'=`\' #1%
  }%
}

\begin{document}

$\vmathrm{`E'}$

$\vmathrm{‘E’}$

\end{document}

