llms.txt
Compare with other sites' llms.txt files →
## Ryft Adaptive Optimization
[Ryft Blog](https://www.ryft.io/blog)
News
# Announcing Ryft Adaptive Optimization
Today, we’re officially introducing **Ryft Adaptive Optimization** \- always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to **5× faster queries**, **10x storage reduction**, and **7x better compaction efficiency** compared to other engines.
## **The Challenge with Manual Table Management**
Managing Iceberg tables is an extremely important yet manual process requiring a lot of testing and constant tweaking to get right.
Data teams need to constantly ensure that each table is maintained and optimized, otherwise performance would plummet and costs would skyrocket.
The challenge is that every table is unique:
1. **Different workload patterns:** Batch processing, CDC streams, real-time ingestion, bulk merges, and large deletes all have distinct optimization needs.
2. **Varying data characteristics**: Wide tables, highly compressible datasets, JSON columns, and high or low cardinality fields each require different approaches.
3. **Unpredictable access patterns**: Query filters, group-bys, and joins constantly evolve, making static optimization strategies obsolete.
This forces teams into endless cycles of monitoring and manual tuning - adjusting compaction schedules, debugging failed maintenance jobs, tweaking compression settings, optimizing file sizes, indexing columns, and reordering data. It's time-consuming, error-prone, and doesn't scale - not a good spend of your time.
## **How Adaptive Optimization Works**
Ryft's engine learns the unique characteristics of each table and automatically adapts optimization strategies accordingly. We analyze three key dimensions:
1. **Workload type:** Streaming workloads need different compaction algorithms and schedules than batch processing.
2. **Data profiling**: We continuously assess data volumes, compression ratios, column counts, and distribution patterns.
3. **Usage analytics:** By collecting query patterns from your compute engines, we make informed decisions about how to optimize each table for maximum impact.
**Looking Under The Covers**
- **Targeted partition compaction:** Only compact partitions that need it, based on file size, age, delete density, and scan frequency.
- **Predictive efficiency scoring:** Evaluate optimization plans upfront and skip work that won't meaningfully improve performance or cost.
- **Intelligent delete file management:** Rewrite equality and position deletes at the right time and for the right partitions, and clean up orphaned delete files.
- **Streaming-aware tiered compaction:** Micro-merges for frequently accessed data, with opportunistic macro-compaction that preserves SLAs.
- **Dynamic resource allocation:** Dynamically allocates resources based on data volume and properties, to achieve the best performance and cost.
- **Smart partition prioritization:** Partitions are prioritized for optimization based on recency, usage, and potential benefit.
- **Metadata optimization:** Reduce query planning overhead by compacting manifests and managing metadata overhead.
- **Dynamic file sizing:** Automatically tune target file sizes based on usage patterns - wider files for scan-heavy fact tables, smaller files for point lookups.
## **Real-World Impact**
### Optimize Only What Matters
For organizations managing hundreds, thousands or tens of thousands of tables, not every table needs constant optimization. Tables that update infrequently don't need constant compaction. Tables created and not updated or rarely used don't need ongoing maintenance.
Ryft’s engine identifies unused tables, optimizes them first and skips unnecessary, ongoing optimizations, ensuring effective use of precious compute resources and budgets.
### CDC Replication Made Simple
Change Data Capture workloads present two unique challenges.
- Many different partitions are frequently updated, making naive "last partition" optimization strategies ineffective.
- Engineers need to monitor and fix commit conflicts when compaction jobs interfere with ongoing table updates.
Ryft monitors all partitions individually, triggering optimization only where changes occurred, while running specialized jobs to minimize commit conflicts and maintain table health.
### Streaming Without the Headaches
Streaming workloads create two persistent problems: endless small files from frequent micro-batches, and commit conflicts between compaction and active writers.
Ryft’s engine detects streaming patterns and adapts its behavior to maintain read performance without interfering with data ingestion, ensuring your streaming pipelines stay healthy and performant.
### **Seamlessly Integrated with Compliance Rules**
For tables requiring GDPR or CCPA compliance, and ongoing retention and tiering, Ryft’s engine coordinates optimization with data retention and compliance cleanup policies. This prevents job collisions and ensures all maintenance tasks are performed efficiently within a unified system.
### **Proven at Scale**
Ryft Adaptive Optimization has been battle-tested in production with several large, multi-petabyte data lakes. The results consistently demonstrate:
- **Up to 5× faster queries** on frequently accessed partitions through targeted compaction and sort rewrites.
- **~7× higher compaction efficiency** compared to other engines, delivering more performance per byte rewritten.
- **Up to 10× storage reduction** through adaptive compression and comprehensive cleanup of snapshots, small files, metadata, and delete files.
Bytes processed by compaction for a 500GB streaming table ingested over a day.
.avif)
Bytes processed by compaction for a 500GB streaming table ingested over a day.

Bytes scanned by queries from a table, before and after enabling Ryft Adaptive Optimization.
## **Ryft Adaptive Optimization is now available for everyone**
Adaptive Optimization is generally available in **Ryft** for all of our customers. Unlock faster queries, predictable SLAs, and lower compute and storage costs, without the manual overhead. [**Contact us**](https://ryft.io/demo) to enable Adaptive Optimization on your highest-value tables and see the impact in your environment.
Table of Contents
[The Challenge with Manual Table Management](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#the-challenge-with-manual-table-management)
[How Adaptive Optimization Works](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#how-adaptive-optimization-works)
[Real-World Impact](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#real-world-impact)
[Optimize Only What Matters](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#optimize-only-what-matters)
[CDC Replication Made Simple](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#cdc-replication-made-simple)
[Streaming Without the Headaches](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#streaming-without-the-headaches)
[Seamlessly Integrated with Compliance Rules](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#seamlessly-integrated-with-compliance-rules)
[Proven at Scale](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#proven-at-scale)
[Ryft Adaptive Optimization is now available for everyone](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization#ryft-adaptive-optimization-is-now-available-for-everyone)
Get the latest posts straight to your inbox
Thank you! Your submission has been received!
## Browse other blogs

News
## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg
Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization.
.avif)
Yuval Yogev
December 8, 2025
December 8, 2025
[blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg)
.avif)
News
## Announcing Ryft Adaptive Optimization
Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines.

Yossi Reitblat
October 23, 2025
September 17, 2025
[blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization)

News
## Unlocking Iceberg management for everyone
Ryft, in many ways, is a story 15 years in the making. Yuval Yogev, Guy Gadon and I went to the same high school, worked together at 8200, building high-scale data infrastructure, and went our separate ways - all to realize that we really enjoy solving complicated data infrastructure problems together with the people we love.

Yossi Reitblat
October 27, 2025
July 8, 2025
[blog](https://www.ryft.io/blog/unlocking-iceberg-management-for-everyone)
## See Ryft in Action
The only solution that automatically maintains and optimizes your Iceberg tables based on usage, no lock-in, no manual tuning
[See Ryft in Action](https://www.ryft.io/see-ryft-in-action)

.avif)
[home](https://www.ryft.io/)
## Ryft Blog Insights
[blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization)
.avif)
News
featured
## Announcing Ryft Adaptive Optimization
Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines.

Yossi Reitblat
October 23, 2025
September 17, 2025

Engineering
## Apache Iceberg V3: Is It Ready?
Apache Iceberg V3 is a huge step forward for the lakehouse ecosystem. The V3 specification was finalized and ratified earlier this year, bringing several long-awaited capabilities into the core of the format: efficient row-level deletes, built-in row lineage, better handling of semi-structured data, and the beginnings of native encryption. This post breaks down the major features, the current state of implementation, and what this means for real adoption.
.avif)
Guy Yasoor
December 16, 2025
December 17, 2025
[blog](https://www.ryft.io/blog/apache-iceberg-v3-is-it-ready)

News
## Announcing Ryft Data Retention & Compliance Enforcement for Apache Iceberg
Today, we’re introducing two new capabilities in Ryft: Automated Data Retention and Data Compliance Enforcement for Apache Iceberg™. These features integrate directly into the Ryft platform to ensure efficient, policy-driven data deletion and compliance, working seamlessly alongside table maintenance and optimization.
.avif)
Yuval Yogev
December 8, 2025
December 8, 2025
[blog](https://www.ryft.io/blog/announcing-ryft-data-retention-compliance-enforcement-for-apache-iceberg)

Engineering
## Streaming with Apache Iceberg: The Operational Problems at Scale
Streaming into Iceberg creates three operational problems most teams don't see coming: small files pile up faster than you can compact them, storage costs climb because you're paying for data you've already replaced, and merges take much longer than they should.
After seeing these problems in real production data lakes, we decided to share more about the causes and possible solutions.
.avif)
Yuval Yogev
December 8, 2025
December 1, 2025
[blog](https://www.ryft.io/blog/streaming-with-apache-iceberg-the-operational-problems-at-scale)

Engineering
## How to Choose an Apache Iceberg Catalog
Apache Iceberg has become the table format of choice for building open data lakehouses. It solves long-standing problems around ACID transactions and engine interoperability. This post covers how to choose an Iceberg Catalog for production.
.avif)
Guy Yasoor
December 8, 2025
November 21, 2025
[blog](https://www.ryft.io/blog/how-to-choose-an-apache-iceberg-catalog)

Engineering
## How to Fix Corrupted Iceberg Tables
In Part 1 and Part 2 of this series, we analyzed two different scenarios that led to Iceberg table corruption - from silent overwrites to inconsistent metadata. Since publishing these posts we have received more requests from people who encountered these situations on how to safely repair those tables. In this post, we’ll focus on the remediation process: identifying what’s affected, how to safely clean it up, and how to prevent further damage.
.avif)
Omer Hadari
November 7, 2025
November 4, 2025
[blog](https://www.ryft.io/blog/how-to-fix-corrupted-iceberg-tables)

Engineering
## Handling Commit Conflicts in Apache Iceberg: Patterns and Fixes
Commit conflicts in Apache Iceberg are one of those problems that seem rare - until you start operating at scale. The first time a long-running compaction job fails after hours of compute, or a CDC pipeline spends half its time retrying commits, you realize this isn’t a corner case. It’s a core operational challenge that directly impacts cost, latency, and reliability.This post covers what commit conflicts are, why they happen, and how to fix them without creating new problems in the process.

Yossi Reitblat
October 23, 2025
October 23, 2025
[blog](https://www.ryft.io/blog/handling-commit-conflicts-in-apache-iceberg-patterns-and-fixes)

Engineering
## Data Retention in Apache Iceberg: Implementation Details and Best Practices
Data retention in Apache Iceberg is one of those critical operations that seems simple until you implement it at scale. Delete old data, save money, stay compliant - straightforward enough. But the implementation details matter, and getting them wrong can mean failed compliance audits, runaway storage costs, or accidentally purging the wrong data.
.avif)
Yuval Yogev
October 23, 2025
October 3, 2025
[blog](https://www.ryft.io/blog/data-retention-in-apache-iceberg-implementation-details-and-best-practices)
.avif)
News
## Announcing Ryft Adaptive Optimization
Today, we’re officially introducing Ryft Adaptive Optimization - always-on, dynamic optimization engine for Apache Iceberg™. Our engine continuously compacts, rewrites, indexes, and reorders data based on how your tables are actually used, delivering up to 5× faster queries, 10x storage reduction, and 7x better compaction efficiency compared to other engines.

Yossi Reitblat
October 23, 2025
September 17, 2025
[blog](https://www.ryft.io/blog/announcing-ryft-adaptive-optimization)

Engineering
## Iceberg Table Corruption and Data Loss in the Wild: Part 2
Data and metadata integrity issues in Iceberg, particularly in streaming workloads, often present similar patterns. In these two posts we covered two similar Iceberg table corruption issues, which manifested in almost the exact same way - data file overwrites. Each time, the underlying reason was different.
.avif)
Omer Hadari
October 21, 2025
August 20, 2025
[blog](https://www.ryft.io/blog/iceberg-table-corruption-and-data-loss-in-the-wild-part-2)

Engineering
## Why Apache Iceberg Finally Unlocks Security Data Lakes
Security data lakes are notoriously hard to build and operate efficiently. Apach