Couchbase 7.0 introduces Scopes and Collections to easily build and deploy microservice based multi-tenant applications. Scopes and Collections allow logical isolation of different types of data, independent lifecycle management and security control at multiple levels of granularity. Application developers will use them to organize and isolate their data. DevOps and Admins will find multiple levels of RBAC (bucket, scope, collection) very powerful to host microservices and tenants at scale. 

The full functionality of this feature is now available in a public beta of release 7.0. A limited functionality Developer Preview was already available in release 6.5 and some of you may have already kicked the tires on that.

What are Scopes and Collections

Scopes and Collections are logical containers within a Couchbase Bucket. Since many are familiar with RDBMS, you can think of Collections as Tables but without any rigidity of data structure since Collections store JSON documents. Scopes are a set of related Collections, similar to what many RDBMSs call “Schema”.

The table below shows the mapping of familiar RDBMS constructs to Couchbase:

RDBMS Mapping

Here is an example of organizing the Couchbase Travel App data with Scopes and Collections:

Scopes and Collections example

How to Create and Use Scopes and Collections

You can create scopes and collections from any of the Couchbase SDKs, couchbase-cli, REST APIs, N1QL, or from the UI. N1QL queries become much simpler and syntactically briefer as you no longer need to qualify different document types using “where type = xxx”. Instead you simply refer to the collection directly. 

In the example shown below, we use N1QL (I executed the N1QL statements using cbq) to create a scope, a couple of collections and indexes and run a simple query that joins two collections.

Create Scope, Collection, Index

Scale of Scopes and Collections

As you may know Couchbase currently allows you to have 30 buckets in a single cluster. Scopes and Collections allow orders of magnitude higher numbers of these entities to be created. 

Since collections are the containers that actually hold the data and scopes organize related collections, you would typically have fewer scopes than collections. Global Secondary Indexes would be created on collections and it would be common for a collection to have multiple indexes.

Keeping these requirements in mind, here is the scale of these entities that you can create in Couchbase 7.0 (a lot of them!):

• Number of Collections allowed per cluster: 1,000
• Number of Scopes allowed per cluster: 1,000
• Number of Global Secondary Indexes allowed per cluster: 10,000

Enabling Multi-Tenancy at Scale 

Modern applications are often written as Microservices with each application comprising several (sometimes 100s) of microservices. With the advent of SaaS and all its advantages, many of these applications are multi-tenant. For a profitable business, it is important to keep the cost (aka TCO) of hosting multiple applications and tenants low, while providing the required isolation and flexibility.

Couchbase Buckets, Scopes and Collections now provide a 3-level containment hierarchy that you can use to map your applications, tenants and microservices.

Key Features for Multi-Tenancy and Microservice Consolidation

Let’s dive into the capabilities offered by Scopes and Collections that make it possible to consolidate tenants and microservices.

Logical Isolation and Indexing

Collections allow you to isolate your data by type while enjoying the flexibility of a JSON document model which can evolve with your application. As shown in the Travel example above the Airline documents go into the Airline collection, the Hotel documents go into the Hotel collection and so forth. 

Each collection can be indexed individually. Different types of documents typically have different indexing needs and collections enable you to define different indexes for each type.

Lifecycle Management at Multiple Levels

Scopes and Collections will allow you to manage and monitor at two new levels in addition to the existing bucket level. 

The provided functionality includes:

• Create and Drop individual collections or scopes
• Set expiration (maximum time to live aka Max TTL) at collection level
• Monitor statistics at scope and collection level. Although a full set of statistics will be available only at bucket level, a subset of statistics will be available at collection and scope levels (e.g. item count, memory used, disk space used, operations per sec).

Fine Grained Access Control with RBAC

One of the most powerful aspects of this offering is the ability to control security at multiple levels – bucket, scope, collection – using Role Based Access Control. This, as you will see later on in this blog, is the key to consolidating hundreds of microservices and/or tenants in a single Couchbase cluster.

Hence, you can now give a user access to a whole bucket or to a scope or to a collection (several out-of-box roles are available, details are available in the documentation).

Fine Grained Replication Control with XDCR

Cross Data Center Replication (XDCR) allows you to control replication at bucket, scope or collection level. You can choose to replicate a whole bucket, a scope or just a collection. You can map to a similarly named entity on the destination or remap to a different entity, along with having the flexibility of advanced filtering already introduced for XDCR in release 6.5 of Couchbase. 

The flexibility provided by XDCR is crucial to mapping microservices and/or tenants to collections and scopes as it allows you to control each microservice and tenant individually.

Backup/Restore at Multiple Levels

In continuation with the theme of providing lifecycle management and access control at multiple levels, we provide the ability to control backup and restore at each of these levels as well. Hence, you can backup (and similarly restore) an individual bucket, or a scope, or a collection along with the ability to filter the data and remap it during restore. 

Do watch out for more detailed how-to blogs on each of these (RBAC, XDCR, Backup/Restore ) in the coming weeks!

Microservices Deployment Using Collections

Microservices architecture recommends that each microservice be simple (ideally single-function), loosely coupled from other microservices (which means have its own data store), and scale independently. Prior to Couchbase 7.0, you could either host each microservice in a separate cluster or a separate bucket. These deployment options have inherent limits on application density and may not always result in full hardware utilization. Now with Couchbase 7.0, you can map each microservice to one or more collections. And since you can have up to a thousand collections, you can have a thousand microservices in a single cluster! 

Let’s look at an example microservice deployment shown below:

Microoservice consolidation

The above example shows two CRM applications (Sales and Customers) deployed in a single cluster. Each of the two applications is mapped to a bucket allowing you to control the overall resource allocation to each application. The Sales application has 2 microservices – Orders and Shipping, and the Customers application has 2 microservices – Individuals and Accounts. We map each of the microservices to a collection each. In this example we did not use scopes. You don’t have to define scopes if you don’t need the extra level of organization (the default scope will be used).

Multi-Tenant Deployment Using Scopes

Multi-tenant applications require varying levels of isolation between tenants and varying levels of resource sharing of the underlying infrastructure. The chosen deployment architecture is a trade-off between isolation and TCO.

Some tenants may require complete physical isolation in which case separate clusters for each tenant may be the right architecture albeit at a high TCO. Couchbase buckets provide partial physical isolation and full security and logical isolation and can be used for separate tenants though with limits of scale and overhead-per-bucket. 

Very often logical and security isolation across tenants is sufficient. With scopes, you get security and logical isolation at more granular levels within a bucket.  You can have thousands of scopes in a single bucket hence enabling you to host thousands of co-operative tenants(not requiring physical isolation) in a single cluster. 

The trade-offs between TCO and isolation levels between different multi-tenant architectures are captured in the table below.

Multi-tenant Architecture Choices

Imagine that the CRM applications in the example above need to now become multi-tenant. You can easily do so by mapping each tenant to a separate scope as shown below (Cust1 and Cust2 are the two different tenants in this example).

Multi-Tenancy with Scopes

Achieving multi-tenancy with scopes is fairly straightforward (without requiring application changes as you can pass a scope context around) and a very scalable and cost effective solution. Collection names need to unique only within a scope. Hence, you can easily deploy multiple scopes with the same collection names as shown above.

Next Steps

I hope you are excited about this new functionality of Scopes and Collections from Couchbase. Below is a list of resources for you to get started and we look forward to your feedback on Couchbase Forums. 

Documentation 

What’s new

Release notes

Get the Beta

Download

Blogs

Introducing Couchbase 7 Beta: Mapping RDBMS to NoSQL

Couchbase Transactions with N1QL