COUCHBASE SWARM SETUP
In this article I would explain how to run couchbase in swarm mode and how to preserve data integrity(by adding nodes with hostname) which can withstand restarts.
When data is persisted through volumes, the new containers which are spun up will not be able to make use the data which is persisted as swarm assigns new IP for start/restarts. Couchbase does not allow hostnames without periods and docker does not allow service names with periods. So,
The challenge is
- How to retain the persisted data across restarts?
- Adding couchbase nodes with hosts names to withstand dynamic IP assignment
Caveat: You should have only one network attached to the service. If more than one network is attached, swarm will assign separate IP address for each network and this will have issues in name resolution while adding new server in couchbase. Same principle applies if you deploy service using stack.
Setup Swarm cluster with 3 nodes.
Pull docker images on the 3 nodes. (or swarm will auto pull when service is deployed)
docker pull sbbabu/couchbase:4.5-community
Create couchbase network on overlay network
docker network create couchbase –d overlay
How to persist data across restarts.
Use docker volume to persist data either with bind mount or volume mounts and set the constraint to deploy one or multiple container as needed. Start container using different volume name for each container so containers has its own volume attached to it.
STEP 1 – Spin up a Couchbase service
docker service create \ --name couchbase1 \ --hostname couchbase-server1 \ --mount type=volume,source=<volumename>,destination=/opt/couchbase/var \ --constraint 'node.hostname == <hostnameofserver1>' \ -p 8091:8091 \ --network couchbase \ -e TZ=Asia/Singapore \ sbbabu/couchbase-community:4.5.0
This will spin up a container with couchbase with user ‘Administrator’ and password ‘password’. Inspecting the Server Nodes tab in couchbase, you will find the server is added with IP for node1. Inspecting the server nodes, we can find the server is added with IP address, (this is not intended), follow the below steps and see how we overcome this.
STEP 2 – Spin up Second container
Spin up second instance of couchbase. Exposing port to host is not required(added for debug purpose only).
docker service create \ --name couchbase2 \ --hostname couchbase-server2 \ --mount type=volume,source=<volumename>,destination=/opt/couchbase/var \ --constraint 'node.hostname == <hostnameofserver2>' \ -p 8892:8091 \ --network couchbase \ -e TZ=Asia/Singapore \ sbbabu/couchbase-community:4.5.0
STEP 3 – Spin up Third container
Spin up third instance of couchbase
docker service create \ --name couchbase3 \ --hostname couchbase-server3 \ --mount type=volume,source=<volumename>,destination=/opt/couchbase/var \ --constraint 'node.hostname == <hostnameofserver3>' \ -p 8893:8091 \ --network couchbase \ -e TZ=Asia/Singapore \ sbbabu/couchbase-community:4.5.0
STEP 4 – Add node2 to cluster
Add the nodes to the cluster. Navigate to the couchbase console and click ‘Add Server’ button. Enter the following Details
Server IP Address*: couchbase2.couchbase Username: Administrator Password: ********
Here server IP address points to the name of the service with the network name as hostname. If you are using different hostname and network, change the values accordingly.
Inspecting the Couchbase console, it should show the new server is added with hostname. Note that, what we see in the ‘Server Node Name’ is the host name and not IP. (unlike the first server)
STEP 5 – Add node3 to cluster
Follow the same step for server3 with different hostname
Navigate to the couchbase console and click ‘Add Server’ button. Enter the following Details
Server IP Address*: couchbase3.couchbase Username: Administrator Password: ********
STEP 6 – Clean up cluster
Now that we have cluster of 3 couchbase, 2 of which are added to the cluster with the hostnames and one is with IP. We will need to change the 1st server IP to host name. To do that, remove the first server from the couchbase admin console and add the server with host name.
From server node click on ‘Remove’ button. (Do rebalance if required)
STEP 7 – Add node1 to cluster
Add server1 to the cluster.
From the Couchbase management console add server one, the same way we did for node2 and node3. Click Add server and enter the host and credentials server1
Server IP Address*: couchbase1.couchbase Username: Administrator Password: ********
Now that we have a 3 node couchbase cluster with server node-names as ‘hostname’.
To test, create a bucket and pump in data. Once the data is loaded, check the count of data on all the nodes. Then, stop(and remove) one of the couchbase container and check that the server is down in the management console. Now start the service of the new container and you can find that the server comes up without any loss in data.