Server side Pagination in Node.js, MongoDB | Mongoose Paginate v2

In previous post, we’ve know how to build Node.js Rest CRUD Apis with MongoDB. This tutorial will show you how to make server side Pagination in Node.js with MongoDB & Mongoose.

More Practice:
Node.js + MongoDB: User Authentication & Authorization with JWT

Deployment: Docker Compose: Node.js Express and MongoDB example

Fullstack with Angular:
Server side Pagination with Node.js and Angular

Clients for this Server:
– React with Material-UI / React with react-table v7
– Angular 8 / Angular 10 / Angular 11 / Angular 12 / Angular 13 / Angular 14
– Vue with Bootstrap / Vuetify

Contents

Server side Pagination
Node.js Pagination with MongoDB overview
Mongoose Paginate v2 for MongoDB pagination
Mongoose Paginate v2 example
Setup Node.js Express Application
Configure MongoDB database
Define the Mongoose Model
Define Mongoose database object
Controller with Pagination
Create Node.js Express API layer
Conclusion
Further Reading
Source Code

Server side Pagination

One of the most important things to make a website friendly is the response time, and pagination comes for this reason. For example, this bezkoder.com website has hundreds of tutorials, and we don’t want to see all of them at once. Pagination means displaying a small number of all, by a page.

If there are only few pages, we can fetch all items and paginate on the client side. It gives us benefit of faster subsequent page loads.

But in case we have large number of pages, make paging on client side will make our client app download all the data at first which might not be needed. So let the server do the work.

Server side pagination is better for:

Large data set
Faster initial page load
Accessibility for those not running JavaScript
Complex view business logic

Node.js Pagination with MongoDB overview

Assume that we have tutorials table in database like this (total 8 documents):

Node.js Express Server will exports API for pagination (with/without filter), here are some url samples:

/api/tutorials?page=1&size=5
/api/tutorials?size=5: using default value for page
/api/tutorials?title=data&page=1&size=3: pagination & filter by title containing ‘data’
/api/tutorials/published?page=2: pagination & filter by ‘published’ status

This is structure of the result that we want to get from the APIs:

{
    "totalItems": 8,
    "tutorials": [...],
    "totalPages": 3,
    "currentPage": 1
}

Read Tutorials with default page index (0) and page size (3):

Indicate page index = 2 but not specify size (default: 3) for total 8 items:

page_0: 3 items
page_1: 3 items
page_2: 2 items

Indicate size = 2 but not specify page index (default: 0):

For page index = 1 and page size = 5 (in total 8 items):

Pagination and filter by title that contains a string:

Pagination and filter by published status:

Mongoose Paginate v2 for MongoDB pagination

To deal with this situation, mongoose-paginate-v2 library provides way to implement pagination with offset and limit properties in the query object that we pass to query methods.

offset: quantity of items to skip
limit: quantity of items to fetch

For example, there are total 8 items.
– { offset: 3 }: skip first 3 items, fetch 5 remaining items.
– { limit: 2 }: fetch first 2 items.
– { offset: 3, limit: 2 }: skip first 3 items, fetch 4th and 5th items.

We’re gonna add plugin to the Schema and use Model paginate() method:

const mongoose = require("mongoose");
const mongoosePaginate = require("mongoose-paginate-v2");
 
const schema = new mongoose.Schema({
  /* tutorial schema definition */
});
 
schema.plugin(mongoosePaginate);
 
const Tutorial = mongoose.model("tutorial", schema);
 
Tutorial.paginate(query, options)
        .then(result => {})
        .catch(error => {});

Mongoose Paginate v2 example

– with offset and limit:

Tutorial.paginate({}, { offset: 3, limit: 2 })
  .then(result => {
    // result:
    /*
    {
      "docs": [
        {
          "title": "bezkoder Tut#4 Rest Apis",
          "description": "Tut#4 Description",
          "published": false,
          "createdAt": "2020-08-24T08:41:21.334Z",
          "updatedAt": "2020-08-24T08:41:21.334Z",
          "id": "5f437d311a98642294ed517a"
        },
        {
          "title": "bezkoder Tut#5 Mongoose",
          "description": "Tut#5 Description",
          "published": false,
          "createdAt": "2020-08-24T08:41:30.067Z",
          "updatedAt": "2020-08-24T08:41:30.067Z",
          "id": "5f437d3a1a98642294ed517b"
        }
      ],
      "totalDocs": 8,
      "offset": 3,
      "limit": 2,
      "totalPages": 4,
      "page": 2,
      "pagingCounter": 3,
      "hasPrevPage": true,
      "hasNextPage": true,
      "prevPage": 1,
      "nextPage": 3
    }
    */
  });

– with select options for getting only title and description on each doc:

Tutorial.paginate({}, { select: 'title description', offset: 3, limit: 2 })
  .then(result => {
    // result:
    /*
    {
      "docs": [
        {
          "title": "bezkoder Tut#4 Rest Apis",
          "description": "Tut#4 Description",
          "id": "5f437d311a98642294ed517a"
        },
        {
          "title": "bezkoder Tut#5 Mongoose",
          "description": "Tut#5 Description",
          "id": "5f437d3a1a98642294ed517b"
        }
      ],
      "totalDocs": 8,
      "offset": 3,
      "limit": 2,
      "totalPages": 4,
      "page": 2,
      "pagingCounter": 3,
      "hasPrevPage": true,
      "hasNextPage": true,
      "prevPage": 1,
      "nextPage": 3
    }
    */
  });

– with query (condition), offset and limit:

Tutorial.paginate({ title: { $regex: new RegExp("mon"), $options: "i" } }, { offset: 1, limit: 2 })
  .then(result => {
    // result:
    /*
    {
      "docs": [
        {
          "title": "bezkoder Tut#5 Mongoose",
          ..
        },
        {
          "title": "bezkoder Tut#8 Mongoose Paginate",
          ..
        }
      ],
      "totalDocs": 3,
      "offset": 1,
      "limit": 2,
      "totalPages": 2,
      "page": 1,
      "pagingCounter": 1,
      "hasPrevPage": true,
      "hasNextPage": true,
      "prevPage": 1,
      "nextPage": 2
    }
    */
  });

– use limit=0 to get only metadata (total documents):

Tutorial.paginate({}, { limit: 0 })
  .then(result => {
    // result:
    /*
    {
      "docs": [],
      "totalDocs": 8,
      "offset": 0,
      "limit": 0,
    }
    */
  });

– with custom return labels:

const myCustomLabels = {
  totalDocs: 'totalItems',
  docs: 'tutorials',
  limit: 'pageSize',
  page: 'currentPage',
  nextPage: 'next',
  prevPage: 'prev',
  totalPages: 'totalPages',
  pagingCounter: 'slNo',
  meta: 'paginator'
};

Tutorial.paginate(condition, { offset: 3, limit: 2, customLabels: myCustomLabels })
  .then((result) => {
    // result:
    /*
    {
      "tutorials": [
        {
          "title": "bezkoder Tut#4 Rest Apis",
          ...
        },
        {
          "title": "bezkoder Tut#5 Mongoose",
          ...
        }
      ],
      "paginator": {
        "totalItems": 8,
        "offset": 3,
        "pageSize": 2,
        "totalPages": 4,
        "currentPage": 2,
        "slNo": 3,
        "hasPrevPage": true,
        "hasNextPage": true,
        "prev": 1,
        "next": 3
      }
    }
    */
  });

Setup Node.js Express Application

First, we create a folder:

$ mkdir node-js-mongodb-pagination
$ cd node-js-mongodb-pagination

Next, we initialize the Node.js App with a package.json file:

npm init

name: (node-js-mongodb-pagination) 
version: (1.0.0) 
description: Server side pagination in Node.js with MongoDB and Mongoose
entry point: (index.js) server.js
test command: 
git repository: 
keywords: nodejs, express, mongodb, rest, api, pagination, mongoose, paginate, server side
author: bezkoder
license: (ISC)

Is this ok? (yes) yes

We need to install necessary modules: express, mongoose, body-parser and cors.
Run the command:

npm install express mongoose mongoose-paginate-v2 body-parser cors --save

You can follow step by step, or get source code in this post:
Node.js, Express & MongoDb: Build a CRUD Rest Api example

The Node.js Express Project contains structure that we only need to add some changes to make the pagination work well.

Or you can get the new Github source code at the end of this tutorial.

Configure MongoDB database

In the app folder, we create a separate config folder for configuration with db.config.js file like this:

module.exports = {
  url: "mongodb://localhost:27017/bezkoder_db"
};

Define the Mongoose Model

In models folder, create tutorial.model.js file like this:

module.exports = mongoose => {
  const Tutorial = mongoose.model(
    "tutorial",
    mongoose.Schema(
      {
        title: String,
        description: String,
        published: Boolean
      },
      { timestamps: true }
    )
  );

  return Tutorial;
};

This Mongoose Model represents tutorials collection in MongoDB database. These fields will be generated automatically for each Tutorial document: _id, title, description, published, createdAt, updatedAt, __v.

{
  "_id": "5e363b135036a835ac1a7da8",
  "title": "Js Tut#",
  "description": "Description for Tut#",
  "published": true,
  "createdAt": "2020-12-02T02:59:31.198Z",
  "updatedAt": "2020-12-02T02:59:31.198Z",
  "__v": 0
}

If you use this app with a front-end that needs id field instead of _id, you have to override toJSON method that map default object to a custom object. So the Mongoose model could be modified as following code:

module.exports = mongoose => {
  var schema = mongoose.Schema(
    {
      title: String,
      description: String,
      published: Boolean
    },
    { timestamps: true }
  );

  schema.method("toJSON", function() {
    const { __v, _id, ...object } = this.toObject();
    object.id = _id;
    return object;
  });

  const Tutorial = mongoose.model("tutorial", schema);
  return Tutorial;
};

And the result will look like this-

{
  "title": "Js Tut#",
  "description": "Description for Tut#",
  "published": true,
  "createdAt": "2020-12-02T02:59:31.198Z",
  "updatedAt": "2020-12-02T02:59:31.198Z",
  "id": "5e363b135036a835ac1a7da8"
}

The most important part is adding mongoose-paginate-v2 plugin to this schema:

module.exports = (mongoose, mongoosePaginate) => {
  var schema = mongoose.Schema(...);

  schema.method("toJSON", function() {...});

  schema.plugin(mongoosePaginate);

  const Tutorial = mongoose.model("tutorial", schema);
  return Tutorial;
};

We’re gonna pass Mongoose and Mongoose Paginate to this Model in the next step.

Define Mongoose database object

Now create app/models/index.js and import mongoose, mongoose-paginate-v2 like the following code:

const dbConfig = require("../config/db.config.js");

const mongoose = require("mongoose");
const mongoosePaginate = require('mongoose-paginate-v2');

mongoose.Promise = global.Promise;

const db = {};
db.mongoose = mongoose;
db.url = dbConfig.url;
db.tutorials = require("./tutorial.model.js")(mongoose, mongoosePaginate);

module.exports = db;

Don’t forget to call connect() method in server.js:

...
const app = express();
app.use(...);

const db = require("./app/models");
db.mongoose
  .connect(db.url, {
    useNewUrlParser: true,
    useUnifiedTopology: true
  })
  .then(() => {
    console.log("Connected to the database!");
  })
  .catch(err => {
    console.log("Cannot connect to the database!", err);
    process.exit();
  });

Controller with Pagination

Earlier on the section above, we’ve known the specific properties in a query object that we pass to paginate() function: offset and limit. But the API receives page and size.

Here is way to calculate:

limit = size
offset = page * size

Notice that we start counting from page 0.

For example, if we want get page number 5 (page=5) and on each page there is 8 records (size=8), the calculations will look like this:

limit = size = 8
offset = page * size = 5 * 8 = 40

So on page 5, there are records from 40 to 47.

Generally, in the HTTP request URLs, paging parameters are optional. So if our Rest API supports pagination, we should provide default values to make paging work even when Client does not specify these parameters.

const getPagination = (page, size) => {
  const limit = size ? +size : 3;
  const offset = page ? page * limit : 0;

  return { limit, offset };
};

By default, 3 Tutorials will be fetched from database in page index 0.

Now the code in tutorial.controller.js will look like this:

const db = require("../models");
const Tutorial = db.tutorials;

const getPagination = (page, size) => {
  const limit = size ? +size : 3;
  const offset = page ? page * limit : 0;

  return { limit, offset };
};

// Retrieve all Tutorials from the database.
exports.findAll = (req, res) => {
  const { page, size, title } = req.query;
  var condition = title
    ? { title: { $regex: new RegExp(title), $options: "i" } }
    : {};

  const { limit, offset } = getPagination(page, size);

  Tutorial.paginate(condition, { offset, limit })
    .then((data) => {
      res.send({
        totalItems: data.totalDocs,
        tutorials: data.docs,
        totalPages: data.totalPages,
        currentPage: data.page - 1,
      });
    })
    .catch((err) => {
      res.status(500).send({
        message:
          err.message || "Some error occurred while retrieving tutorials.",
      });
    });
};

// Find all published Tutorials
exports.findAllPublished = (req, res) => {
  const { page, size } = req.query;
  const { limit, offset } = getPagination(page, size);

  Tutorial.paginate({ published: true }, { offset, limit })
    .then((data) => {
      res.send({
        totalItems: data.totalDocs,
        tutorials: data.docs,
        totalPages: data.totalPages,
        currentPage: data.page - 1,
      });
    })
    .catch((err) => {
      res.status(500).send({
        message:
          err.message || "Some error occurred while retrieving tutorials.",
      });
    });
};

// other CRUD functions

Create Node.js Express API layer

The final part is to use the Controller methods in tutorial.routes.js with Express Router.
There are 2 important endpoints: /api/tutorials/ and /api/tutorials/published.

module.exports = app => {
  const tutorials = require("../controllers/tutorial.controller.js");

  var router = require("express").Router();

  // Retrieve all Tutorials
  router.get("/", tutorials.findAll);

  // Retrieve all published Tutorials
  router.get("/published", tutorials.findAllPublished);

  ...

  app.use('/api/tutorials', router);
};

Conclusion

In this post, we have learned how to create Server side Pagination in Node.js with MongoDB database using Express, Mongoose. We also see that Mongoose Paginate v2 supports a great way to make pagination and filter methods without need of boilerplate code.

React Pagination Client that works with this Server:
– React Table Pagination using react-table v7
– React Pagination with API using Material-UI

Angular Client working with this server:
– Angular 8 Pagination example
– Angular 10 Pagination example
– Angular 11 Pagination example
– Angular 12 Pagination example
– Angular 13 Pagination example
– Angular 14 Pagination example

Or Vue Client:
– Vue Pagination example (Bootstrap)
– Vuetify Pagination example

Happy learning! See you again.

Source Code

You can find the complete source code for this tutorial on Github.

4 thoughts to “Server side Pagination in Node.js, MongoDB | Mongoose Paginate v2”

Antonin Massuelle says:

May 31, 2022 at 4:03 pm

Thank you so much, without your tutorial it would have taken me ages to manage to implement this.

Everything is perfectly explained and detailed !
Taahir says:

August 13, 2021 at 4:49 pm

Awesome article! Thank you very much.
Irina says:

March 10, 2021 at 12:01 am

Thank you very much! Your article helped me to solve my problem!!!
Serena says:

October 13, 2020 at 9:42 am

The descriptions and examples above all seek to understand the subject. Many thanks! Keep writing guy!

Comments are closed to reduce spam. If you have any question, please send me an email.