← Back to Curriculum

MongoDB Sharding

📚 Lesson 12 of 15 ⏱️ 45 min

MongoDB Sharding

45 min

MongoDB sharding distributes data across multiple servers (shards) to enable horizontal scaling. Sharding is essential for handling datasets that exceed the capacity of a single server or for distributing load across multiple machines. Sharded clusters consist of mongos routers (query routers), config servers (metadata storage), and shard servers (data storage). Understanding sharding enables you to scale MongoDB to handle very large datasets and high throughput.

Mongos routers are lightweight processes that route queries and write operations to appropriate shards. Applications connect to mongos routers instead of shard servers directly. Mongos routers use metadata from config servers to determine which shard contains data for a given query. Multiple mongos routers can be deployed for high availability and load distribution. Understanding mongos enables you to design scalable MongoDB architectures.

Config servers store metadata about the sharded cluster, including which shard contains which data ranges (chunks). Config servers are critical for cluster operation and should be deployed as a replica set for high availability. The metadata includes shard configuration, chunk distribution, and balancer settings. Understanding config servers enables you to manage and troubleshoot sharded clusters.

Shard keys determine how data is distributed across shards. MongoDB divides data into chunks based on the shard key and distributes chunks across shards. Choosing an appropriate shard key is critical for even data distribution and query performance. Poor shard key selection can lead to uneven data distribution (chunk imbalance) or inefficient queries (scatter-gather queries). Understanding shard keys enables optimal sharding strategies.

The balancer automatically redistributes chunks across shards to maintain even data distribution. When chunks become too large or unevenly distributed, the balancer migrates chunks between shards. The balancer runs in the background and can be enabled or disabled as needed. Understanding the balancer enables you to manage data distribution in sharded clusters.

Sharding considerations include choosing appropriate shard keys, monitoring chunk distribution, managing balancer operations, and understanding query routing. Sharding adds complexity to MongoDB deployments but enables scaling beyond single-server limitations. Understanding sharding enables you to design MongoDB architectures that scale horizontally.

Key Concepts

  • Sharding distributes data across multiple servers for horizontal scaling.
  • Mongos routers route queries to appropriate shards.
  • Config servers store cluster metadata and configuration.
  • Shard keys determine how data is distributed.
  • The balancer maintains even data distribution across shards.

Learning Objectives

Master

  • Understanding sharded cluster architecture
  • Setting up and configuring sharded clusters
  • Choosing appropriate shard keys
  • Managing sharded cluster operations

Develop

  • Understanding horizontal scaling strategies
  • Designing scalable MongoDB architectures
  • Managing large-scale MongoDB deployments

Tips

  • Enable sharding: sh.enableSharding('database') to enable for a database.
  • Shard collection: sh.shardCollection('db.collection', { shardKey: 1 }).
  • Check sharding status: sh.status() to see cluster configuration.
  • Monitor balancer: sh.getBalancerState() to check balancer status.

Common Pitfalls

  • Choosing poor shard keys, causing uneven data distribution.
  • Not monitoring chunk distribution, missing imbalance issues.
  • Not understanding query routing, causing inefficient scatter-gather queries.
  • Not configuring config servers as replica set, risking metadata loss.

Summary

  • Sharding enables horizontal scaling for large datasets.
  • Mongos routers, config servers, and shards work together in sharded clusters.
  • Shard key selection is critical for performance and data distribution.
  • Understanding sharding enables scaling MongoDB beyond single-server limits.

Exercise

Set up and configure MongoDB sharding.

// Start config servers
mongod --configsvr --port 27019 --dbpath /data/configdb

// Start shard servers
mongod --shardsvr --port 27018 --dbpath /data/shard1
mongod --shardsvr --port 27020 --dbpath /data/shard2

// Start mongos router
mongos --configdb localhost:27019

// Connect to mongos and add shards
mongosh --port 27017

// Add shards to cluster
sh.addShard("localhost:27018")
sh.addShard("localhost:27020")

// Enable sharding for database
sh.enableSharding("myApp")

// Shard a collection
sh.shardCollection("myApp.users", { email: 1 })

// Check sharding status
sh.status()

// Check balancer status
sh.getBalancerState()

// Start/stop balancer
sh.startBalancer()
sh.stopBalancer()

// Check chunk distribution
use config
db.chunks.find({ ns: "myApp.users" })

// Add new shard
sh.addShard("localhost:27022")

// Remove shard
sh.removeShard("localhost:27020")

// Check shard statistics
db.users.getShardDistribution()

// Query with specific shard
db.users.find({ email: "alice@example.com" }).explain("queryPlanner")

// Update shard key
// Note: This requires careful planning and data migration
db.users.updateMany(
  {},
  { $set: { newShardKey: "$email" }}
)

Exercise Tips

  • Choose shard keys with good cardinality and distribution.
  • Monitor chunk sizes: db.chunks.find({ ns: 'db.collection' }) to check distribution.
  • Use compound shard keys for better distribution: { field1: 1, field2: 1 }.
  • Plan shard key changes carefully: changing shard keys requires data migration.

Code Editor

Output