We had to import a huge collection into our production system's mongo db. It was configured as cluster of three nodes - primary, secondary and arbiter. The collection would start importing and show progress gradually. But after a while it would get stuck at 40% and not increase even after a long time.
The status of the nodes were PRIMARY, RECOVERING and ARBITER. After digging deep into the "RECOVERING" status - I found out that the secondary was way behind in replication of data. The reason was, our system had scheduled indexing process that would constantly keep writing / deleting into multiple collections, every few hours. This cluster had been up for almost a year or more. Over the period of time, the lag had added up so much that while the PRIMARY was writing every few hours, secondary was still replicating something that was already deleted by the primary.
Because of the above situation, mongo restore was not working. Restore waits for write and replication to complete. But secondary was not up for replication.
Since it was on production, we could not bring down the mongo systems by restarting it. So we had to find an alternative : To bring the SECONDARY up to date. I had read somewhere that if a node had to be added to an existing cluster, the replication would be quite fast - a matter of few minutes given a decent size of databases. To bring up a new server and install mongo and configure everything was a tedious task. After googling, I found out that you can clear the data folder of a non-primary node without bringing the database down. So I stopped the SECONDARY mongo service, deleted the data folder, and restarted the service again. Replication took place within a minute and the status changed to "SECONDARY". After this, I restored the database and it completed in the normal stipulated time.
The status of the nodes were PRIMARY, RECOVERING and ARBITER. After digging deep into the "RECOVERING" status - I found out that the secondary was way behind in replication of data. The reason was, our system had scheduled indexing process that would constantly keep writing / deleting into multiple collections, every few hours. This cluster had been up for almost a year or more. Over the period of time, the lag had added up so much that while the PRIMARY was writing every few hours, secondary was still replicating something that was already deleted by the primary.
Because of the above situation, mongo restore was not working. Restore waits for write and replication to complete. But secondary was not up for replication.
Since it was on production, we could not bring down the mongo systems by restarting it. So we had to find an alternative : To bring the SECONDARY up to date. I had read somewhere that if a node had to be added to an existing cluster, the replication would be quite fast - a matter of few minutes given a decent size of databases. To bring up a new server and install mongo and configure everything was a tedious task. After googling, I found out that you can clear the data folder of a non-primary node without bringing the database down. So I stopped the SECONDARY mongo service, deleted the data folder, and restarted the service again. Replication took place within a minute and the status changed to "SECONDARY". After this, I restored the database and it completed in the normal stipulated time.
No comments:
Post a Comment