Understanding Couchbase Mobile (sync)

This post is also available in: Portuguese (Brazil)

We’ve seen in the last article how to use Couchbase Lite to persist data locally. But the real power of it stands out with its synchronisation mechanism, which brings the benefits of a fast local database together with the data exchange demand of today’s applications.

This article is an introduction to the infra-structure that acts as a synchronisation end-point for Couchbase Lite database. It will show you, how all the pieces fit together, so that you are loaded with the concepts needed to create a real-life mobile application, that functions both stand-alone (off-line), and online (pushing/pulling data when connected). This is not limited to mobile devices (smartphones), but can be also used in embedded computers which eventually get disconnected but need to continue operating. It is expected that you’re familiar with Couchbase Lite, know what git (versioning tool) is, and most importantly, that you are looking forward to learning new things to take your app to the next level.

Let’s get started.

So far, our application had a local database (Couchbase Lite), where we save and query data:

Screen Shot 2014-10-28 at 5.43.08 PM

As we saw, the mobile app’s code can interact with the information either via model objects or with the documents themselves. We now want the data to eventually reach a server so that when the same user access the app from another device, he will have his information restored in his mobile device, as well as  off-line changes pushed back to the server.

The first idea that comes to mind is to simply store the whole database in the server, as a “dummy” backup solution. It does not take long to realise that this approach would not scale, is not efficient, and does not handle version conflicts.

Like in git or rsync, we want to synchronise only the changes, which shall be applied or not according to the validity of the new data: the term used for this in Couchbase Mobile is data replication. When another user connects to the same backend, we want only the documents belonging to him to be synchronised, and the term used for this is data routing.

Most well elaborated applications involving user data needs to deal with the following:

  • Authentication (is the user who he/she claims to be?)
  • Authorisation (who is allowed to modify what?)
  • Data validation (are changes valid?)

So before we begin plugging our app database with “the cloud”, we should understand how such problems are solved in couchbase mobile.

In the typical request-response RESTful service, the back-end code will be in charge of validating/authorising the input and performing the required actions:

Screen Shot 2014-10-28 at 2.15.11 PMWe can apply the user credentials (think Basic HTTP auth) to the requests, so that the server knows whether this request can proceed or not (authentication). Then the back-end code will check if the http payload data is valid / consistent before pushing them to a database. Encapsulate all of it in a SSL connection and everything will be fine (that is, while your connection works).

Notice how in the request – response model, data is “routed” by the different API end-points:

Screen Shot 2014-10-29 at 2.34.57 PM

The request URI path will trigger a specific request handler in the server.

Even though couchbase does not enforce a document schema, that doesn’t mean you want corrupted / non-sense data in your database ;). So the same sort of “protections” and routings described above must be possible to be applied during synchronisation with couchbase mobile.

In order for Couchbase not to depend on an active connection, what each data represents and the respective routing are described in the document’s content itself (instead of being separated by API calls):

Screen Shot 2014-10-29 at 2.36.08 PM

The back-end can then check the contents of the documents, looking for “type”, and so it will have the information to continue with processing, validating and routing of the data.

This way the app gains the advantages of a distributed system: allowing it to operate in the document locally (off-line), and when eventually connected to the server, the doc’s meta-data will indicate where it belongs to and what it means. This design is not a replacement for the existing http rest api’s (although it sometimes can be), but is instead an independent complement to it (make sure to use the right tool for the right job).

For this reason, after we setup a server to replicate our data, information will be exchanged and verified at the Document level, in the same sense that git merges code line-by-line.

Every document should be validated (according the app’s business logic) and routed, to succeed with synchronisation in the back-end. The minimum validation I can think of would be to know if the user sending/requesting the document is authorised to do so (is not messing with somebody else’s data).

In Couchbase mobile, the component responsible for such tasks is a single binary program, called Sync Gateway, that sits right between the database server and your mobile application, acting as a “controller”.

Expanding on the previous drawing, we now have this:

Screen Shot 2014-10-28 at 7.05.49 PM

We will connect our mobile devices’ database to this gateway end-point to handle the documents sync. The gateway will be configured to decide whether a document update is valid or not to be pushed/pulled to the database.

To summarise: in the mobile app we take care of creating/updating the documents, in the sync gateway we provide the rules for authorising a document creation/update in the database, and the sync gateway itself is backed-up by the database server (Couchbase Server) which acts as a storage container for these documents.

So the components involved are:

  • Couchbase Lite:  Embedded lightweight, document-oriented (NoSQL), syncable database engine.
  • Couchbase Server: Distributed (shared-nothing architecture) NoSQLdocument-oriented database.
  • Sync Gateway: Interface between Couchbase Lite and Database Server (enables Couchbase Server to act as a replication endpoint for Couchbase Lite). Manages HTTP-based data access for mobile clients and handles access control and data routing, so that a single large Couchbase Server cluster can manage data for multiple users.

As mentioned, the sync gateway should also know if the client requesting the sync is who he claims he is (authentication), therefore it needs to do user management:

Such users are managed from the server side through a special port in the gateway (via http api requests), and during replication, it will cross-check if the credentials provided by Couchbase Lite match his user/session values. At the moment of writing the following authentication methods are available: HTTP Basic Auth, or Session Based, which will be seen in more detail during implementation.

In the next article, “Configuring your own Couchbase mobile sync backend“, we put the theory shown here into action.