You are working on an API endpoint to create a new user and you will require some data along with the request like firstname, lastname, age and birthdate for the user you are about to create. Obviously, passing
Sally as value for age or
53 for birthdate won't get things rolling in the right direction. You really don't want bad data making its way through your application so what do you do? The answer is Data Validation.
If you have ever used an ORM when building your Node application such as Sequelize, Knex, Mongoose (for MongoDB), etc, you.
In this tutorial, we will learn how we can use the Joi validation module to validate data at the request level. You can learn more about how to use Joi and the supported schema types by checking out the API Reference. At the end of this tutorial, we should be able to do the following:
Table of Contents
- Create validation schema for the request data parameters
- Handle validation errors and give appropriate feedback
- Create a middleware to intercept and validate requests
Getting Started
We will pretend we are building a mini-school portal and we want to create API endpoints to be able to:
- add new students and teachers
- set login credentials for teachers
- make fee payments for students
We will create a very simple REST API for this tutorial using Express just to test our Joi schemas. To begin, jump to your command line terminal and run the following command to setup a new project and install the required dependencies.
npm init -y npm install --save express body-parser morgan joi lodash
Create a new file named
app.js in your project root directory to setup the Express app. Here is a simple setup for the application.
/* app.js */ // load app dependencies const express = require('express'); const logger = require('morgan'); const bodyParser = require('body-parser'); const Routes = require('./routes'); const app = express(); const port = process.env.NODE_ENV || 3000; // app configurations app.set('port', port); // load app middlewares app.use(logger('dev')); app.use(bodyParser.json()); app.use(bodyParser.urlencoded({ extended: false })); // load our API routes app.use('/', Routes); // establish http server connection app.listen(port, () => { console.log(`App running on port ${port}`) });
It is worth noting that we have added
body-parser middlewares to the request pipeline of our app. These middlewares basically fetch and parse the body of the current HTTP request for
application/json and
application/x-www-form-urlencoded requests and makes it available in the
req.body of the request's route handling middleware.
From our application setup, we specified that we are fetching our routes from a
routes.js file. Let's quickly create the file on our project root directory with the following content:
/* routes.js */ const express = require('express'); const router = express.Router(); // generic route handler const genericHandler = (req, res, next) => { res.json({ status: 'success', data: req.body }); }; // create a new teacher or student router.post('/people', genericHandler); // change auth credentials for teachers router.post('/auth/edit', genericHandler); // accept fee payments for students router.post('/fees/pay', genericHandler); module.exports = router;
Finally, add a
start script to the
scripts section of your
package.json file. It should look like this:
{ ... "scripts": { "start": "node app.js" }, ... }
Let's go ahead to run our app to see where we've gotten to and to ensure everything is fine.
npm start
Mine is running on port 3000, so testing the API endpoints using Postman works as expected. Here is a sample screenshot of testing the
POST /people endpoint on Postman.
A Simple Example
I think a simple example will help give us an idea of what we hope to achieve. You may not understand everything in this example at the moment, but you definitely will by the time you are done with this tutorial.
About the ExampleIn this example, we will create validation rules using Joi to validate an email, phonenumber and birthday for a request to create a new user. If the validation fails, we send back an error. Otherwise, we return the user data.
Let's add a test route to the
app.js file. Add the following code snippet to the
app.js file for our simple example.
/* app.js */ /** * CODE ADDITION * * Defines a POST /test route for our simple example * * It must come after the following line: * app.use('/', Routes); */ app.post('/test', (req, res, next) => { // require the Joi module const Joi = require('joi'); // fetch the request data const data = req.body; // define the validation schema const schema = Joi.object().keys({ // email is required // email must be a valid email string email: Joi.string().email().required(), // phone is required // and must be a string of the format XXX-XXX-XXXX // where X is a digit (0-9) phone: Joi.string().regex(/^\d{3}-\d{3}-\d{4}$/).required(), // birthday is not required // birthday must be a valid ISO-8601 date // dates before Jan 1, 2014 are not allowed birthday: Joi.date().max('1-1-2004').iso(), }); // validate the request data against the schema Joi.validate(data, schema, (err, value) => { // create a random number as id const id = Math.ceil(Math.random() * 9999999); if (err) { // send a 422 error response if validation fails res.status(422).json({ status: 'error', message: 'Invalid request data', data: data }); } else { // send a success response if validation passes // attach the random ID to the data response res.json({ status: 'success', message: 'User created successfully', data: Object.assign({id}, value) }); } }); }); /* CODE ADDITION ENDS HERE */
Here, I have tried to keep things as simple as possible. I have added some comments to describe what each line does. Don't worry if everything is not clear to you at the moment. They will be clear to you as you progress in the tutorial. Now we will go ahead to test the example route.
Start the app again by running
npm start from your terminal and head to Postman to test the example route
POST /test. Here is a demo from my machine.
Introduction to Joi
While this tutorial is not a substitute to the actual documentation and usage guide for the Joi package, we'll try as much as possible to explore some important parts of the Joi API as required for our sample project.
Validating Anything
Joi uses schemas to define validation rules and constraints for data. You can use
Joi.any() to create a simple schema that validates for any data type. This means that any value checked against the schema will be valid. The
any() schema is the base schema which every other Joi schema types inherit from.
const any = Joi.any();
You can specify more validation constraints to the base schema to control the kind of values that are considered valid. Since each constraint returns a schema instance, it is possible to chain several constraints together via method chaining to define more specific validation rules. Here are some very useful validation constraints that can be attached to the base schema:
allow()
Whitelists value or set of values that will always be valid before applying other rules. It can take values as its arguments or an array of values.
valid()
Whitelists value or set of values as the only valid value(s) before applying other rules. It can take values as its arguments or an array of values.
invalid()
Blacklists value or set of values that will never be valid. It does the direct opposite of
allow(). It can take values as its arguments or an array of values.
required()
Prevents the schema from allowing
undefinedas value.
optional()
The schema can allow
undefinedas value. The schema however, will not allow a value of
null. This is the default behaviour.
raw()
This outputs the original untouched value instead of the casted value.
strict()
This enables strict mode - which prevents type casting for the current key and any child keys. In non-strict mode, the values are casted to match the specified validation schema where possible. This is the default behaviour.
default()
This sets a default value if the original value is
undefined. In its simplest form, it takes as its first argument the value to use as default. See the API Reference for a detailed documentation of the
default()constraint.
Having seen these constraints, we can quickly create a schema that matches only the strings
cat and
dog as follows:
const catOrDog = Joi.any().valid(['cat', 'dog']).required();
Validating Strings
The
Joi.string() schema allows you to validate only string values. It inherits from the
any() schema, hence all the constraints we saw earlier can be used with it. However, it provides some more constraints for validating strings more effectively. Here are some common ones:
min(),
max(),
length()
These are used to control the minimum length, maximum length or fixed length of the string. They each take as first argument, an
integerthat specifies the length limit.
ip(),
guid(),
uri(),
creditCard()
These are used to validate email addresses, IP addresses, GUIDs, URIs or credit card numbers (using the Luhn Algorithm) respectively. They each take an optional options
objectas first argument. See the API Docs for detailed information about the supported options.
alphanum(),
hex(),
base64()
These are used to restrict the valid strings to alphanumeric, hexadecimal or base64-encoded strings respectively.
regex(),
replace()
These allow you to specify a custom regular expression that the string must match for it to be considered valid. They each take a
RegExpliteral as first argument.
regex()can also take an optional options
objectas second argument.
replace()is useful if you want to replace some parts of the matched string with another string. It takes the replacement
stringas second argument.
lowercase(),
uppercase(),
insensitive()
These are used to force a case or ignore the case (for
insensitive()) on the string during validation. Note that case conversion may happen on the original value except when strict mode is enabled.
Here are some string schema examples:
// accepts only valid lowercase email addresses const email = Joi.string().email().lowercase().required(); // accepts alphanumeric strings at least 7 characters long const password = Joi.string().min(7).alphanum().required(); // accepts an optional US phone number of the format (XXX) XXX-XXXX or XXX-XXX-XXXX // it defaults to 111-222-3333 if undefined // Note: that optional is the default behaviour when there is no explicit required() const phone = Joi.string().regex(/^(\(\d{3}\) |\d{3}-)\d{3}-\d{4}$/).default('111-222-3333');
Validating Numbers
The
Joi.number() schema makes it more convenient to validate numeric values. It also inherits from the
any() schema, hence all the constraints on the
any() can be used with it. Here are a few more constraints you can use with the
number() schema:
min(),
max(),
greater(),
less()
These are used to put a limit on the range of numbers that are considered valid. They each take as first argument, an
integerthat specifies the limit.
precision()
This specifies the maximum number of decimal places for numbers that are considered valid. It takes an
integeras first argument that specifies the maximum decimal places.
positive(),
negative()
These are used to restrict the valid numbers to only positive or negative numbers respectively.
integer()
This allows only integers (no floating point) to be considered as valid values. In non-strict mode, some type conversion may occur on the value if possible.
Here are some examples:
// accepts only positive numbers with max of 2 decimal places const amount = Joi.number().positive().precision(2).required(); // accepts an optional positive integer that must be greater than 13 const age = Joi.number().greater(13).integer().positive();
Validating Dates
Date values can be validated using the
Joi.date() schema. It also inherits from the
any() schema, hence all the constraints on the
any() can be used with it. Here are some additional constraints you can use with the
date() schema:
min(),
max()
These are used to put a limit on the range of dates that are considered valid. They each take as first argument, a valid
Datevalue that specifies the limit. You can also pass the string
nowto use the current date as limit.
iso()
This requires the date value to be in valid ISO 8601 format.
timestamp()
This requires the date value to be a timestamp interval from Unix Time. It takes an optional
stringas first argument that determines the type of timestamp to validate with. The allowed values are
unixand
javascript. Defaults to
javascript.
Here are some examples:
// accepts only dates since Jan 1, 2017 in ISO 8601 format // ISO 8601 format: 2017-01-01 const since = Joi.date().min('1-1-2017').iso().required(); // accepts an optional JavaScript style timestamp (with milliseconds) const timestamp = Joi.date().timestamp('javascript');
Advanced Schemas
Now that we have a good grasp of basic Joi schemas, we will go a step further to see how we can define more advanced schemas.
Object Schemas
Most of the data we will be working with are usually JavaScript
objects that have
keys and
values. We should as a result be able to validate a full object with keys and values. Joi provides several ways to specify schema for an object. Let's see a quick example of a schema that specifies validation rules for an object containing
lastname and
age.
// Method 1: (Object Literal) const schema1 = { firstname: Joi.string().required(), lastname: Joi.string().required(), age: Joi.number().integer().greater(10) }; // Method 2: (Using Joi.object()) const schema2 = Joi.object({ firstname: Joi.string().required(), lastname: Joi.string().required(), age: Joi.number().integer().greater(10) }); // Method 3: (Using Joi.object().keys()) const schema3 = Joi.object().keys({ firstname: Joi.string().required(), lastname: Joi.string().required(), age: Joi.number().integer().greater(10) });
Though each of the above methods creates an object schema, there are some subtle differences. It is recommended that you create object schemas using
Joi.object() or
Joi.object().keys(). When using any of these two methods, you can further control the keys that are allowed in the object using some additional constraints, which will not be possible to do using the object literal method. Here are some additional constraints:
pattern()
This allows you specify validation rules for keys that match a given pattern. It takes two arguments. The first is a
RegExpliteral for the pattern expression unknown keys will be matched against. The second is the Joi schema with which to validate unknown keys that match the pattern. Here is an example:
// this matches unknown keys like created_at, last_updated_at, etc // and ensures that they have a date value in the past (in ISO 8601 format) const schema = Joi.object({}).pattern(/^([a-z]+)(_[a-z]+)*?_at$/, Joi.date().max('now').iso().required())
and(),
nand(),
or(),
xor()
These allow you specify a relationship between one or more keys. Each accept key names or an array of key names with which to establish the relationship. For
and(), if one of the specified keys is present, then the others must also be present. For
nand(), if one of the specified keys is present, then the others cannot be present. For
or(), at least one of the specified keys must be present. For
xor(), at least one of the specified keys must be present but never all the keys together. See the API Reference for usage examples.
with(),
without()
These allow you to specify keys that must or must not appear together with a particular key. They both accept two arguments. The first is the key that is used as a reference. The second is an
arrayof keys that must or must not appear with the reference key. See the API Reference for usage examples.
Using Key References
When defining validation rules for a key, it is possible to make reference to another key on the same object schema. To specify a reference to a key, you use
Joi.ref(). It takes as first argument the name of the key to reference or a variable in the schema validation context object. Dot notation is allowed for nested key names.
Here is an example of using key references:
// we use a reference to the min key on max // to always ensure that max is greater than min const schema1 = Joi.object({ min: Joi.number().integer().positive().required(), max: Joi.number().integer().greater(Joi.ref('min')).required() }); // we use a reference to the password key on confirmPassword // to always ensure that password and confirmPassword are exactly the same const schema2 = Joi.object({ password: Joi.string().min(7).required().strict(), confirmPassword: Joi.string().valid(Joi.ref('password')).required().strict() });
Alternative Schemas
Sometimes, you may want a value to be either a string or number or something else. This is where alternative schemas come into play. You can define alternative schemas using
Joi.alternatives(). It inherits from the
any() schema, so constraints like
required() can be used with it. There are two validation constraints for creating alternative schemas.
try()
Using
try()we can specify an array of alternative schemas that will be accepted as valid values. It is equivalent to using just an ordinary array literal containing the alternative schemas. Here is an example:
// a positive integer e.g 32 const number = Joi.number().integer().positive().required(); // a numeric string that represents a positive integer e.g '32' const stringNumber = Joi.string().regex(/^\d+$/).required(); // schema1 and schema2 are equivalent const schema1 = Joi.alternatives().try(number, stringNumber); const schema2 = [number, stringNumber];
when()
Using
when()allows you specify alternative schemas based on another key. It takes the key as first argument and an options
objectspecifying the conditions as second argument. Here is a quick example:
// a positive integer e.g 32 const number = Joi.number().integer().positive().required(); // a numeric string that represents a positive integer e.g '32' const stringNumber = Joi.string().regex(/^\d+$/).required(); // when useString is true, use either number or stringNumeric schema // otherwise use only number schema const schema = Joi.object({ value: Joi.alternatives().when('useString', { is: true, then: [number, stringNumber], otherwise: number }), useString: Joi.boolean().default(true) });
Creating the API Schemas
We are now up to speed with Joi schemas. We will go ahead to create the validation schemas for our API routes. Create a new file named
schemas.js in the project route directory and add the following code snippet to it.
/* schemas.js */ // load Joi module const Joi = require('joi'); // accepts name only as letters and converts to uppercase const name = Joi.string().regex(/^[A-Z]+$/).uppercase(); // accepts a valid UUID v4 string as id const personID = Joi.string().guid({version: 'uuidv4'}); // accepts ages greater than 6 // value could be in one of these forms: 15, '15', '15y', '15yr', '15yrs' // all string ages will be replaced to strip off non-digits const ageSchema = Joi.alternatives().try([ Joi.number().integer().greater(6).required(), Joi.string().replace(/^([7-9]|[1-9]\d+)(y|yr|yrs)?$/i, '$1').required() ]); const personDataSchema = Joi.object().keys({ id: personID.required(), firstname: name, lastname: name, fullname: Joi.string().regex(/^[A-Z]+ [A-Z]+$/i).uppercase(), type: Joi.string().valid('STUDENT', 'TEACHER').uppercase().required(), sex: Joi.string().valid(['M', 'F', 'MALE', 'FEMALE']).uppercase().required(), // if type is STUDENT, then age is required age: Joi.when('type', { is: 'STUDENT', then: ageSchema.required(), otherwise: ageSchema }) }) // must have only one between firstname and lastname .xor('firstname', 'fullname') // firstname and lastname must always appear together .and('firstname', 'lastname') // firstname and lastname cannot appear together with fullname .without('fullname', ['firstname', 'lastname']); // password and confirmPassword must contain the same value const authDataSchema = Joi.object({ teacherId: personID.required(), email: Joi.string().email().lowercase().required(), password: Joi.string().min(7).required().strict(), confirmPassword: Joi.string().valid(Joi.ref('password')).required().strict() }); // cardNumber must be a valid Luhn number const feesDataSchema = Joi.object({ studentId: personID.required(), amount: Joi.number().positive().greater(1).precision(2).required(), cardNumber: Joi.string().creditCard().required(), completedAt: Joi.date().timestamp().required() }); // export the schemas module.exports = { '/people': personDataSchema, '/auth/edit': authDataSchema, '/fees/pay': feesDataSchema };
Here, we have created schemas for our API endpoints and exported them in an object with the endpoints as keys. A quick summary of the schemas is as follows:
personDataSchema
- id is required and must be a valid UUID v4 string.
- type must be either
TEACHERor
STUDENT. It can be in any case since we are not using strict mode. However, the final output will be converted to uppercase.
- sex must be one of
M,
F,
MALEor
FEMALE. It can be in any case since we are not using strict mode. However, the final output will be converted to uppercase.
- age is optional if type is
TEACHERbut required if type is
STUDENT. The age can be an integer or a string made up of an integer followed by
y,
yror
yrs. The age integer must be 7 and above. When the value is a string, the non-integer parts are stripped off in the final value. So possible values can be:
10,
'10',
'10y',
'10yr',
'10yrs', etc.
- If fullname is specified, then firstname and lastname must be ommitted. If firstname is specified, then lastname must also be specified. One of either fullname or firstname must be specified.
authDataSchema
- teacherId is required and must be a valid UUID v4 string.
- email is required and must be a valid email string.
- password is required and must be a string with a minimum of 7 characters.
- confirmPassword is required and must be the exact same string as password.
feesDataSchema
- studentId is required and must be a valid UUID v4 string.
- amount is required and must be a positive number greater than
1. If a floating point number is given, the precision is truncated to a maximum of
2. Since we are not in strict mode, a number with precision greater than 2 will still be accepted but will be truncated.
- cardNumber is required and must be a valid Luhn algorithm compliant number. Every card number is a valid Luhn algorithm compliant number. For test purposes, use
4242424242424242as card number.
- completedAt is required and must be a JavaScript timestamp.
Schema Validation Middleware
Let's go ahead to create a middleware that will intercept every request to our API endpoints and validate the request data before handing control over to the route handler.
Create a new folder named
middlewares in the project root directory and then create a new file named
SchemaValidator.js inside it. The file should contain the following code for our schema validation middleware.
/* middlewares/SchemaValidator.js */ const _ = require('lodash'); const Joi = require('joi'); const Schemas = require('../schemas'); module.exports = (useJoiError = false) => { // useJoiError determines if we should respond with the base Joi error // boolean: defaults to false const _useJoiError = _.isBoolean(useJoiError) && useJoiError; // enabled HTTP methods for request data validation const _supportedMethods = ['post', 'put']; // Joi validation options const _validationOptions = { abortEarly: false, // abort after the last validation error allowUnknown: true, // allow unknown keys that will be ignored stripUnknown: true // remove unknown keys from the validated data }; // return the validation middleware return (req, res, next) => { const route = req.route.path; const method = req.method.toLowerCase(); if (_.includes(_supportedMethods, method) && _.has(Schemas, route)) { // get schema for the current route const _schema = _.get(Schemas, route); if (_schema) { // Validate req.body using the schema and validation options return Joi.validate(req.body, _schema, _validationOptions, (err, data) => { if (err) { // Joi Error const JoiError = { status: 'failed', error: { original: err._object, // fetch only message and type from each error details: _.map(err.details, ({message, type}) => ({ message: message.replace(/['"]/g, ''), type })) } }; // Custom Error const CustomError = { status: 'failed', error: 'Invalid request data. Please review request and try again.' }; // Send back the JSON error response res.status(422).json(_useJoiError ? JoiError : CustomError); } else { // Replace req.body with the data after Joi validation req.body = data; next(); } }); } } next(); }; };
Here, we have loaded Lodash alongside Joi and our schemas into our middleware module. We are also exporting a factory function that accepts one argument and returns the schema validation middleware. The argument to the factory function is a
boolean value which when
true, indicates that Joi validation errors should be used, otherwise a custom generic error is used for errors in the middleware. It defaults to
false if not specified or a non-boolean value is given.
We have also defined the middleware to only handle
PUT requests. Every other request methods will be skipped by the middleware. You can also configure it if you wish, to add other methods like
DELETE that can take a request body.
The middleware uses the schema that matches the current route key from the
Schemas object we defined earlier to validate the request data. The validation is done using the
Joi.validate() method, with the following signature:
- data is the data to validate which in our case is
req.body.
- schema is the schema with which to validate the data.
- options is an
objectthat specifies the validation options. Here are the validation options we used:
- callback is a callback
functionthat will be called after validation. It takes two arguments. The first is the Joi
ValidationErrorobject if there were validation errors or
nullif no errors. The second argument, is the output data.
Finally, in the callback function of
Joi.validate() we return the formatted error as a JSON response with the
422 HTTP status code if there are errors, or we simply overwrite
req.body with the validation output data and then pass control over to the next middleware.
Now we can go ahead to use the middleware on our routes. Modify the
routes.js file as follows:
/* routes.js */ const express = require('express'); const router = express.Router(); const SchemaValidator = require('./middlewares/SchemaValidator'); // We are using the formatted Joi Validation error // Pass false as argument to use a generic error const validateRequest = SchemaValidator(true); // generic route handler const genericHandler = (req, res, next) => { res.json({ status: 'success', data: req.body }); }; // create a new teacher or student router.post('/people', validateRequest, genericHandler); // change auth credentials for teachers router.post('/auth/edit', validateRequest, genericHandler); // accept fee payments for students router.post('/fees/pay', validateRequest, genericHandler); module.exports = router;
Let's go ahead to run our app to test what we've done. These are sample test data you can use to test the endpoints. You can edit them however you wish. For generating UUIDv4 strings, you can use the Node UUID module or an online UUID Generator.
POST /people
{ "id": "a967f52a-6aa5-401d-b760-35eef7c68b32", "type": "Student", "firstname": "John", "lastname": "Doe", "sex": "male", "age": "12yrs" }
{ "teacherId": "e3464323-22c1-4e31-9ac5-9bde207d61d2", "email": "email@domain.com", "password": "password", "confirmPassword": "password" }
POST /fees/pay
{ "studentId": "c77b8a6e-9d26-428a-9df1-e852473f886f", "amount": 134.9875, "cardNumber": "4242424242424242", "completedAt": 1512064288409 }
Here are screenshots of testing the API endpoints on Postman.
Conclusion
In this tutorial, we seen how we can create schemas for validating a collection of data using Joi and how to handle request data validation using a custom schema validation middleware on our HTTP request pipeline. For a complete code sample of this tutorial, checkout the joi-schema-validation-sourcecode repository on Github.