AWS › Amazon Kinesis & Data Streaming

How does the partition key affect how records are distributed across shards in Kinesis?

Intermediate

Answer

Each record written to a Kinesis stream includes a partition key, which Kinesis hashes to determine which specific shard should store that record, meaning that choosing a partition key with high variety, such as a unique device ID, spreads records evenly across all available shards, while a partition key with only a few possible values can overload just one or two shards, creating an uneven and inefficient distribution of data.

aws kinesis put-record --stream-name my-stream --data 'sensor reading' --partition-key 'device-12345'

Real-world example An IoT platform uses each device's unique serial number as the partition key when sending sensor data to Kinesis, ensuring an even spread of traffic across all shards regardless of how many devices are actively reporting data.

Common follow-ups: What happens if you use the same partition key for every record?;Can a single shard become a bottleneck even with a good partition key strategy?

Amazon DynamoDB;Auto Scaling Groups