Is it possible to create a custom keyboard and upload it to the App store? I am working with someone who teaches an endangered language, of interest to a fairly small group. Many of the people in that group use only phones, and don’t have laptops. The language has a lot of special characters to represent the sounds of the language — that is, it has its own alphabet — but some of those characters are not available in any of the standard languages available by default on OSX.


Is it possible to design a custom keyboard and define which keys will input which character?
If so, is it possible to upload such a keyboard to the app store to make it available to people (he wants to make it free)?
Yes, lots of people have done this, you can find many custom language keyboards in the app store.  You should make sure there is not one there already by searching the store for the language you are talking about.

Have a look at the Keyman app.  It already has lots of keyboards and I think offers ways to create new ones. 

Keyman 

For developer info see iOS Custom Keyboard