Configuring your own couchbase mobile sync backend

Configuring your own couchbase mobile sync backend

This article explains the main steps involved in setting up a working sync gateway end point with user management. It is assumed you have basic knowledge of POSIX-like system services administration and networking. It is also assumed that you know the design of a Couchbase Mobile Application (explained in Understanding Couchbase Mobile article).

By the end of the article, you should have your app replicating Couchbase Lite documents to your own server.

To summarise, the work can be broken down to the following steps, starting with the server towards the mobile client:

  1. Install Couchbase server;
  2. Create a container in the Couchbase server for our application documents;
  3. Install Sync Gateway, point it to the respective database server container;
  4. Configure the gateway with an authentication system;
  5. Programming the sync gateway’s document validation / routing;
  6. Testing: Configure the mobile app to synchronise with the sync gateway.

Steps 1 to 4, which is the focus of this article, provides the basic synchronisation infra-structure for your application, and is mostly about administration work. Step 5 will actually contain app/business-specific logic, which most probably will be written incrementally together with the development of the mobile app (client). I will dedicate a separate article with focus on data validation / routing. In the demo application of this article, we apply no document enforcements, and instead just worry first about getting the data pushed and pulled to test the infrastructure we setup.

 

Step 1 and 2: Preparing the Couchbase Server

Meet Couchbase Server. We will not go into deep details on Couchbase server here, but will instead just set up and explain the minimum so that we can sync with our application. Other optimisations can be made as your user base / demand, and knowledge about distributed databases grows.

Download and install Couchbase server in a computer, paying attention to the minimum requirements (i.e., that it has a sufficient amount of RAM memory). This is a database prepared for big clusters, so if your resources are limited during development, one trick I recommend is to add extra space using a swap file (Linux). Other than that, it’s as easy as a mysql installation for small projects.

Here is the link for the official download page: http://www.couchbase.com/nosql-databases/downloads

Current versions of couchbase provides a web administration panel which can be accessed at port 8091. Make sure you go through it to perform the initial setup. This port will also be the access point for the sync gateway.

What you need to know if you’re new to the database:

Couchbase Server has the concepts of “Data Buckets“. It is nothing more than a container to organise documents, much like what a database instance is in Couchbase Lite. One Bucket is really like one database in the RDBMS terminology. Just another name.

We will need one bucket for our application, so we now create one using the database’s web interface. There are a few options when creating a bucket, but for now the defaults will suffice; We just give it a name: “app_bucket”.

 

Step 3: Install Sync Gateway, point it to the created “app_bucket”

Now download and run the Sync Gateway. Since it is just a binary, you may want to manage it using something like supervisor (linux). Another trick I use is the “screen” program, so that it does not terminate if I want to close the shell access. If you do install it as a package, for example in ubuntu, the binary can be found at /opt/couchbase-sync-gateway/bin/sync_gateway.

The sync gateway uses a configuration file to know what it should do. This file is passed as a command line argument when starting the service. Let’s check it’s contents:

The “interface” option defines where the sync gateway will be listening: this is where the mobile client app will connect to.

The “adminInterface” is a privileged port used for tasks related to authorisation and authentication. Therefore it should be blocked from the outside (e.g. firewall).

The “log” option configures what should be logged to the standard output by the program. Other flags useful for debugging flags are: [“CRUD”, “CRUD+”, “HTTP”, “HTTP+”, “Access”, “Cache”, “Shadow”, “Shadow+”, “Changes”, “Changes+”].

Now the most interesting part: inside the parameter “databases“, we define our app’s endpoint(s). In the example we call it “sync_gateway“. That means later, when we want to setup our mobile application, we’ll provide the couchbase lite SDK an URL similar to the following:

#define kSyncUrl @”http://yourapp.com:4984/sync_gateway

Inside the sync_gateway “end-point” parameter, we find other three items:

server” : Location of your Couchbase server and port number.

bucket” : The data bucket name created in step 1 / 2.

sync” : Responsible for document authorisation, validation, routing. We will implement / explain this in step 6, for now we leave the default which does no verification.

When you have the configuration file ready, pass it on when starting the sync gateway:

Step 4: Configure the gateway with an authentication system

At the time of this writing (Sync Gateway v.1.0.2, Couchbase Lite 1.0.3), the following authentication methods are available:

  • HTTP Basic Auth
  • Session based:
    • Custom Service (backed by own user database, or LDAP, etc)
    • Others: Facebook, Persona.

We will now setup authentication with a custom auth app server. The advantage of having a custom auth is that one does not need to rely in a third party organisation (such as Facebook). The disadvantage is that you have to code the service yourself to talk with the sync gateway and the respective calls from the mobile app. Also the end-user might need to create another user/password combination if you’re not developing against an existing source (like LDAP).

Anyhow, the same concept is applied to the other session-based auth mechanisms, so if you follow this explanation, using another authentication mechanism with be even easier (the third party server provides the SDK to get the session keys, and takes care of the credentials).

The sync gateway provides RESTful administration API’s which allows us to manage users and sessions in the gateway. Session keys provided by the gateway can then be used by the mobile (client) to authenticate his synchronisation requests. Following is a sample workflow; The credential verification itself can happen in any desired way, e.g.: LDAP or own local mysql db, etc:

Screen Shot 2014-10-30 at 10.22.48 AM

  1. The client requests his session key to the authentication service by providing his credentials;
  2. If the credentials are correct, the authentication service asks the sync gateway for a session for this user;
  3. The sync gateway responds to the auth server with the session key;
  4. The Auth server forwards the obtained session key back to the client;
  5. The client can now establish replication directly with the sync gateway using the received session key.

One important note for the step “2a”: every new user will need to be first created in the sync gateway before a session key can be generated. The app can check the response (number 3), and if not existent PUT the newUser, and then repeat the POST /_session request.

Below is a sample python/flask web service request handler to perform this:

Tip: Debugging the gateway during development

The sync gateway does not store anything on itself; Instead, it uses the database bucket we provided for all its storage needs. That said, we can use curl to interact with it and then inspect the database to see what happens when using these sync gateway’s API’s, see:

Creating an user “by hand”

Now if we navigate to our Couchbase Server’s administration panel, we can enter the bucket, and check the data by clicking in “View Document”. Let’s check how the database document looks like for our newly created user (the gateway named it:  “_sync:user:firstUser:”)

Getting the session for the newly created user:

response from the gateway:

The same thing we did by hand using curl, can be done by any scripting language on your authentication server.

 

 

Step 5 – Configure the sync gateway (validation function)

With all the infra-structure in place, we can test everything by synchronising a few documents. For that, you can modify the sync function of the sync gateway’s configuration to this:

Create a document in your mobile app that contains the “username” property at its root. You can then save and sync the document and check in your database’s bucket if the data passed through. Now even if you delete the mobile client app database, if the replication starts with a document with the same user as in doc.username, the document created previously should be pulled back to the device.

A next article will go into details about the sync function.

 

 

Step 6: Testing: Configure the mobile app to synchronise with the sync gateway.

At this stage, we finished setting up the backend infrastructure needed to provide replication to our client / mobile app. There is one last “wire” we need to plug to make the show begin, and that is, point couchbase lite to our now ready-to-use sync gateway.

It’s time to take a break from the server, and go back to the mobile app side. Go ahead and test your service by following the article “Connecting Couchbase Lite database to Sync Gateway“.

Note about SSL (Secure Sockets Layer)

There are to ways to add SSL to the connection between the mobile client connection to the sync gateway. One is to install a proxy in front of the gateway to handle SSL (e.g.: nginx). In this case, everything is transparent for the sync gateway, and the only change in the mobile client side is to add the “s” to the http:// URL.

The second way is to activate SSL encryption in the gateway itself, instructions for that can be found in this link.

  • Adam Wilson

    Is it possible to connect the Sync Gateway to another database e.g. CouchDB or Cloudant?
    Thanks for the article!

    • Thiago Alencar

      Well, it’s an open source software so theoretically one can do anything 🙂

      But officially, no. My answer is backed by an employee’s answer you can find here : Couchbase Mobile Google Groups

    • By the way, just a small note in case someone else reads this: although not compatible with e.g. MongoDB, it will work with CouchDB (but missing the features of the Sync Gateway).

    • if you mean substitute Sync Gateway by CouchDB than yes,
      if you mean directing Sync Gateway to couchDB than no

  • Eric Bolikowski

    Thank you so much for this article. It was very helpful in helping me (for the first time) configure Sync Gateway and a Android client running CBlite.

  • Célio Silva

    Thanks for this article, it’s very helpful. One question though, as I’ve understood the interface will require the user to be authenticated but the adminInterface will never require authentication. Is that correct ? Is it up to you to secure the admin path between your server application and sync gateway ? TKX!

    • Hello Célio!
      That is correct, the admin port must be isolated from the public network, and secure behind your firewalls.
      Im glad to help!
      BR!

      • Célio Silva

        TKX Thiago, keep up the good work!