Mongo db replica set

Usually, our systems are configured such that only servers in the same network can access it. So all servers in the same network were able to access the mongo db in its network. However, due to some twisted reason, we had one machine outside this network that needed access. So we opened the IP of this lonesome machine to the mongo db server.

One unfortunate day, everything breaks. Mongo connection timeout logs everywhere! But login to mongo machine and open mongo terminal, it works. And the dbs are intact. Check the security groups to see if the servers have access to mongo, its all fine. Then what is the issue?

We look at logs more carefully now. There is only one IP being logged..and this was the secondary. In the code, we were always connecting to one configured IP, which was the previous PRIMARY. The mongo nodes had switched its states. So, we made the code change to read IPs using the replica set name. This way, it reads the current PRIMARY IP, at all times. A change is made in all components to read all private IPs in the connection url. (Except the lonesome one, of course)

MongoClient mongoClient = new MongoClient( new MongoClientURI( "mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=myReplicaSet"));

On the lonesome machine's application we configured all the public IPs, with the replicaSet name and deploy. All applications except the one on lonesome machine is not able to connect to mongo. On looking at the logs, we find that it still is using private IPs, although the configured IPs were public. Strange...

Then we do an rs.status() on the mongo to find that all the "name" fields of the configured nodes have private IPs. There! Even if you configure with public IPs in your application, it uses the replica set and gets the names and tries to connect with those IPs. So we had to keep it the same as before, configure using a single public IP. But this doesn't ensure that you're application will work when the primaries switch. (For us it was still ok, since we did not have a replica set on prod)

Mongo db restore problem

We had to import a huge collection into our production system's mongo db. It was configured as cluster of three nodes - primary, secondary and arbiter. The collection would start importing and show progress gradually. But after a while it would get stuck at 40% and not increase even after a long time. 

The status of the nodes were PRIMARY, RECOVERING and ARBITER. After digging deep into the "RECOVERING" status - I found out that the secondary was way behind in replication of data. The reason was, our system had scheduled indexing process that would constantly keep writing / deleting into multiple collections, every few hours. This cluster had been up for almost a year or more. Over the period of time, the lag had added up so much that while the PRIMARY was writing every few hours, secondary was still replicating something that was already deleted by the primary. 

Because of the above situation, mongo restore was not working. Restore waits for write and replication to complete. But secondary was not up for replication.  

Since it was on production, we could not bring down the mongo systems by restarting it. So we had to find an alternative : To bring the SECONDARY up to date. I had read somewhere that if a node had to be added to an existing cluster, the replication would be quite fast - a matter of few minutes given a decent size of databases. To bring up a new server and install mongo and configure everything was a tedious task. After googling, I found out that you can clear the data folder of a non-primary node without bringing the database down. So I stopped the SECONDARY mongo service, deleted the data folder, and restarted the service again. Replication took place within a minute and the status changed to "SECONDARY". After this, I restored the database and it completed in the normal stipulated time.