Migrate From MongoDB to Couchbase with N1QL

Customers often tell us they’re preparing to migrate from MongoDB to Couchbase. They come, in part, because they’re tired of the problems they’ve experienced learning how to query MongoDB. Couchbase with N1QL provides a better alternative, especially for scaling up modern applications.

Unfortunately, with MongoDB, migration options can be limited. One I wrote about recently involves moving from MongoDB with Mongoose to Couchbase with Ottoman.  The core of that tutorial was around using two different ODM tools in Node.js that shared the same APIs, making the transition almost seamless.  However, what if you’re not using an ODM in MongoDB and have no interest in using one?

This time around we’re going to take a look at moving a Node.js application that uses MongoDB and the MongoDB query language to Couchbase with N1QL.  In short, N1QL is a technology that lets you run SQL queries against complex JSON data.  This makes it not only easy to use, but very clean in the application layer.  More information on N1QL can be found in the Couchbase Developer Portal.

We’re going to be using the same example problem used in the previous article, but it is alright if you haven’t seen it.  Everything here will be started with a clean slate.

The Requirements

There are a few requirements that must be met to make sure you are successful with this guide.  They can be seen as follows:

• • MongoDB 3.4+
  • Couchbase 4.5+
  • Node.js 6.0+

Because we will be seeing both the MongoDB and Couchbase equivalent, you should have both databases available to you.  If you’re already a MongoDB developer, a lot may be familiar to you, but that is your call.

Understanding the NoSQL Data Model

Before developing the RESTful API with either technology, it is a good idea to first understand the data model that will be used.

While both MongoDB and Couchbase are document databases, they are not entirely the same.  MongoDB stores data as BSON while Couchbase stores data as JSON.  From a modeling perspective, it won’t really matter to us.

Take the data model for a school where you have students and you have courses.  For every course offered at the school, you might have a document that looks like the following:

Each course will maintain a list of all the students that have been registered.  In this case, the list will consist of id values that reference other documents.  We’re establishing our own document relationships.

For every student at the school, they may have a NoSQL document that looks like the following:

Notice that the above document is similar to how we modeled our courses.  Each student will maintain a list of all the courses that they are registered to.  These courses are ids that reference the appropriate course document.

There are a hundred different ways we can model our documents, this is just one example.  For more information on data modeling, check out this documentation.

Now that we have a model for our documents, we can focus on building an API with each technology.

Developing an API with the MongoDB Query Language

While you might have your own MongoDB application code, we’re going to create one from scratch so the migration remains very easy to understand.

Let’s start by creating a fresh project from our Command Prompt or Terminal:

The above commands will create a new package.json file in your current working directory and install the necessary framework and database dependencies.

At the end of the day, we want our MongoDB project to have the following structure:

All interaction with the database will be done from each of the models and all API routing and interaction with the client will be done from the routes files.  Bootstrapping the application and connecting to the database will be done from the app.js file.

Creating a MongoDB Database Model within the Application

So let’s take a look at one of our database models.  Open the project’s models/course.js file and include the following code:

There is a lot happening in the above database model.  We need to break it down to make it easy to understand.  The more we understand, the easier the migration process to Couchbase will be.

When we wish to save a document, we have the save method:

Using the open database we can insert a single document into the desired collection using the insertOne function.  In this function we pass data which can be a JavaScript object of any complexity that we wish to save.  Once saved, the result will be returned to the parent method that called it.

Now what if we want to update a document that already exists?  In particular, what if we want to add a student to an already existing course?

For this particular example, we can update the entire students array that exists in the document:

The above code will use the updateOne method to lookup a particular document by id and replace the students array with a new one that we provide via the updateStudents function.

Nothing too difficult so far and no real stress added to the developer.

This is where things change.  When we save data we are saving an array of id values.  This is not data we want to see in query operations.  Instead we want to populate or expand these id values into their document equivalents:

To make this possible, the array must first be flattened via an $unwind and joined via a $lookup operation.  It doesn’t stop there because we want our results to be formatted in the same fashion, only replacing the ids for objects.  Because of this we have to do further flattening, grouping, and manipulations.

A full explanation of joining data in MongoDB versus joining data in Couchbase can be seen in a thorough article I wrote previously on the subject.

Long story short, the more complicated your data, the more complicated your aggregation query will be.  For a flexible data model, this becomes not very flexible for an application developer.

Let’s take a quick look at our other model, the one that will manage student data.  Open the project’s models/student.js file and include the following JavaScript code:

Pretty much the same rules apply to the above model versus the model we saw representing course data.  This is because the two document models were very similar to begin with.  Flat for the most part with an array of id values.

This brings us to the API endpoints that make use of these database methods.

Creating the RESTful API Routes for the Application

This is the easy part, and actually the most consistent between the two database technologies because it isn’t database dependent.

Let’s have a look at the project’s routes/courses.js file:

In the above code we have three API routes.  From an external client we’ll be able to list all courses, find a particular course, or save a new course.

Depending on which endpoint is hit, the appropriate database model function will run.  Not much heavy lifting is done in the routes, which based on their name, are only meant for routing.

The other routing file is going to be a bit different.  Open the project’s routes/students.js file and include the following JavaScript code:

Like with the other route, there is a lot going on, but most of that is near identical.  Where are differences are falls in the /student/course endpoint that is responsible for adding courses to a student and students to a course.

When hitting this endpoint we first get the student and course information based on the passed id value.  They both need to exist, otherwise we’ll throw an error.  If they both exist, we’ll push the student id into the students array for the course document and the course id into the courses array of the student document.  Then we’ll call our update method and return a result to the end user.

Bringing it Together and Bootstrapping the Application

The API is ready to go at this point.  We just need to bootstrap the Node.js application and connect to the database.  This is the easy part.

Open the project’s app.js file and include the following code:

In the above code we are importing each of the dependencies that we downloaded and initializing Express Framework.  Before we start serving the API, we need to establish a connection to MongoDB.  Once the connection is established, the routes are connected and the application starts serving.

At this point the RESTful API can be reached at http://localhost:3000 and tested with popular tools like Postman or Fiddler.

Developing an API with Couchbase and N1QL

So we have an example to work with when it comes to MongoDB and Node.js.  That example used the MongoDB query language when communicating with the database.

Now we’re going to take that application and move it to Couchbase.  We’re doing this because not only does Couchbase scale better and have better performance, the query language is far simpler and easier to maintain within an application.

Like with the MongoDB application, we’re going to start from scratch, even though much of what we see will be identical.  From the Command Prompt or Terminal, execute the following:

The above commands should look familiar.  We’re creating a package.json file and installing our dependencies, but instead of MongoDB we’re using Couchbase.

The project will hold the same structure as seen in the previous project.  It should look like this:

The same logic as seen previously will end up in each of these files.  The difference being the syntax for Couchbase.

Creating a Couchbase Database Model within the Application

Starting with the same order, we’re going to create our database model functions.  As mentioned earlier, we’re going to be using N1QL which is an extreme highlight of Couchbase because it allows you to write SQL queries.  These queries are executed on the database and not within your Node.js application.

Open the project’s models/course.js file and include the following code:

In the above code we have the same set of database related functions that we saw in the MongoDB example.  Inside each of these functions are N1QL queries.  Not just N1QL queries, but parameterized N1QL queries to help fight off SQL injection attacks.

Take a look at the getAll method that was previously super complicated in the MongoDB version:

This time around we have a simple query that includes a subquery.  Should our data needs become more complex, the query can be altered without gaining significant size or complexity.

So let’s look at our other database model.  Open the project’s models/student.js file and include the following JavaScript code:

Look familiar?  The two database models are similar because the two document models are similar.  Again, should the complexity change, the application layer will still be easy to manage with N1QL queries.

Because the API routes don’t have any dependencies on the database, the code between the MongoDB application and the Couchbase application can be shared.  To be clear, I’m referring to the files found in the routes directory.

Bootstrapping the Application and Connecting to Couchbase

With the endpoints in place and the database models communicating to Couchbase through N1QL, we can finish off the application.

Open the project’s app.js file and include the following JavaScript code:

In the above code we are importing the dependencies that we had installed and initializing Express Framework.  Then we are establishing a connection to the database, bringing our routes together, and starting the Node.js server.

Because we are using N1QL, don’t forget to create at least one index on your Couchbase Bucket.  It can be as simple as the following:

At this point you should be able to access your application in the same fashion as MongoDB.  Visit http://localhost:3000 from your web browser or with a tool like Postman or Fiddler.

Conclusion

You just saw how to take a Node.js application that uses MongoDB and the MongoDB Query Language and convert it to Couchbase with N1QL instead.  This is an alternative to the Mongoose to Ottoman ODM solution that I wrote about previously.

So why would you want to switch from MongoDB to Couchbase?  Well Couchbase is a lot faster and easier to scale, but N1QL is also incredibly simple when working with complex data.  You’ll be able to reduce your code drastically and keep it more maintainable.  Check out the Couchbase vs. MongoDB query tutorial I wrote about in terms of joining data between the two databases.

For more information on how to use Couchbase in a Node.js application:

• • Check out the Couchbase Developer Portal.
  • Try one of the Couchbase developer IDEs—JetBrains, VSCode—that we have plugins for.