Hey Rockset User!

With today's DynamoDB connector (aka v1) Rockset uses the scan functionality in order to bring in your initial table data into a Rockset collection while using your DynamoDB RCUs. While this works well for tables up to 10s of GB in size, it can often timeout for large tables (1TB+) since DynamoDB scan functionality and ability to parallelize the scan is limited. With the new DynamoDB connector, Rockset will leverage DynamoDB's ability to export a table to S3, and can rapidly ingest data at a much higher rate, making the initial table data load much faster.

Once this will be enabled for your Rockset organization, you'll need to create a new integration that will require additional permissions, mainly related to S3, due to the nature of how this new connector works. Last, Rockset will also need a designated S3 bucket to export your tables to. Once the initial load phase is finished you can safely delete the exported data from that S3 bucket.

All we ask is that you provide us with feedback on how easy/hard the setup is, and how well it performed against your expectations. It'll be a bonus if you already use our current v1 version of the connector and can compare and contrast your experience. Your input is highly valuable.

Note: There's no change in the CDC (Change Data Capture)/Streaming ingest functionality. Rockset will continue to consume all changes to your DynamoDB tables as they come.