Redshift Serverless PoC

Summary

• Reserving Redshift nodes is the best option given specific usage patterns.
• Reserved nodes are 30% cheaper than on-demand ones.
• Serverless doesn't show stable and production-ready results: faced with multiple query failures during data refresh.
• Moreover, Serverless will definitely cost more with the same usage patterns.

Comparison Matrix

Detailed Redshift cost estimations

Pricing: https://aws.amazon.com/redshift/pricing/

On-demand nodes:

Compute

1. ra3.4xlarge = $3.418 per Hour
2. 3.418 * 2 * 24 = $164,064 per day
3. 164,064 * 30 = 4921,92 per month per cluster

Storage

1. $0.024 per GB per month
2. $0.024 * 2000 = $48 per month

Reserved nodes (No Upfront)

Compute

1. $1,746.60 per month per node
2. $3492 per month per cluster

Storage

1. 0.024 * 2000 = $48 per month

Serverless

Option 1:

1. RPU capacity = 64Stockholm: $0.374 per RPU hour
2. Capacity used - 64 RPUs
3. Daily charges = 64 * 0,374 * 8 hours = $192
4. Monthly charges = $5760

Option 2:

1. RPU capacity = 128Stockholm: $0.374 per RPU hour
2. Capacity used - 128 RPUs
3. Daily charges = 128 * 0,374 * 6 hours = $287
4. Monthly charges = $8616

Redshift Usage Statistics (daily pattern)

Capture PROD cluster statistics: number of working queries by groups during the whole day.

• Execution patterns by user group during a workday

• Total number of minutes a cluster is running on a given day

Which is ~11 hours

Serverless causes errors to dbt jobs

I have run total DWH refresh 3 times and every time there were queries which failed with an error.

This is unacceptable as dbt job is failing and subsequent models are skipped.

Error message:

[0m07:08:42.613995 [error] [MainThread]:   server closed the connection unexpectedly
[0m07:08:42.614260 [error] [MainThread]:    This probably means the server terminated abnormally
[0m07:08:42.614519 [error] [MainThread]:    before or while processing the request.