llms.txt
Compare with other sites' llms.txt files →# OLake
> Open-source data replication and Apache Iceberg table maintenance platform, moving data from databases, Kafka and S3 into Apache Iceberg on object storage.
OLake is a high-performance, open-source data platform with two products. OLake Go is an EL (Extract and Load) engine that replicates data from sources like PostgreSQL, MySQL, MongoDB, Oracle, MSSQL, IBM Db2, Apache Kafka, and S3 into Apache Iceberg tables or Parquet files on object storage, supporting full refresh, incremental sync, and Change Data Capture (CDC) with stateful, resumable syncs. OLake Fusion automates Apache Iceberg table maintenance, including compaction of small files, and in OLake's compaction benchmark (https://olake.io/blog/iceberg-compaction-spark-vs-fusion-benchmark/) finished in 27m 2s against 55m 47s for Apache Spark. In OLake's published ingestion benchmark (https://olake.io/docs/benchmarks/ingestion/), OLake Go loaded 4.01 billion Postgres rows at 580,113 rows per second and ran CDC at 55,555 rows per second (January 2026). Both products are open source under the Apache 2.0 license.
OLake is built for data engineers, data architects, and developers building or scaling data infrastructure. Key use cases include setting up CDC pipelines into Apache Iceberg, migrating databases to cloud object storage, keeping Iceberg tables compacted and query-ready, and enabling real-time analytics on fresh data. Its focus on open table formats avoids vendor lock-in: OLake writes standard Iceberg tables compatible with catalogs such as AWS Glue, Hive Metastore, and REST catalogs (Nessie, Polaris, Unity Catalog), and queryable from engines including Trino, Spark, Snowflake, Databricks, and ClickHouse.
## Products & Core Documentation
- [OLake Go Overview](https://olake.io/docs/): Getting started with OLake Go, the open-source EL engine for replicating databases and Kafka to Apache Iceberg, with supported sources, destinations, and capabilities.
- [OLake Fusion Overview](https://olake.io/docs/fusion/getting-started/overview/): Getting started with OLake Fusion for automated Apache Iceberg table maintenance and compaction.
- [OLake Quickstart Guide](https://olake.io/docs/getting-started/quickstart/): Start using OLake Go with a single Docker command and access the UI.
- [Configure Your First Ingestion Job](https://olake.io/docs/getting-started/creating-first-pipeline/): Step-by-step guide to creating a first OLake Go replication job.
- [OLake Playground](https://olake.io/docs/getting-started/playground/): Try OLake Go in a sandboxed playground environment.
- [OLake Go vs Fivetran, Debezium, Estuary & Airbyte: PostgreSQL Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=postgres): Ingestion throughput benchmark comparing OLake Go against Fivetran, Debezium, Estuary, and Airbyte Cloud for PostgreSQL to Apache Iceberg full-load and CDC sync.
- [OLake Go vs Fivetran, Debezium, Estuary & Airbyte: MongoDB Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=mongodb): Ingestion throughput benchmark comparing OLake Go against Fivetran, Debezium, Estuary, and Airbyte Cloud for MongoDB to Apache Iceberg sync.
- [OLake Go vs Fivetran, Debezium, Estuary & Airbyte: MySQL Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=mysql): Ingestion throughput benchmark comparing OLake Go against Fivetran, Debezium, Estuary, and Airbyte Cloud for MySQL to Apache Iceberg sync.
- [OLake Go vs Fivetran, Debezium, Estuary & Airbyte: Oracle Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=oracle): Ingestion throughput benchmark comparing OLake Go against Fivetran, Debezium, Estuary, and Airbyte Cloud for Oracle to Apache Iceberg sync.
- [OLake Go vs Apache Flink: Kafka Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=kafka): Ingestion throughput benchmark comparing OLake Go against Apache Flink for Kafka to Apache Iceberg sync.
- [OLake Go vs Fivetran, Debezium, Estuary & Airbyte: MSSQL Benchmark](https://olake.io/docs/benchmarks/ingestion/?tab=mssql): Ingestion throughput benchmark comparing OLake Go against Fivetran, Debezium, Estuary, and Airbyte Cloud for MSSQL to Apache Iceberg sync.
## Install & Configure
- [Install with Docker Compose (UI)](https://olake.io/docs/install/olake-ui/): Deploy OLake Go with its web UI using Docker Compose.
- [Install with Docker (CLI)](https://olake.io/docs/install/docker-cli/): Run OLake Go from the command line using Docker.
- [Install on Kubernetes with Helm](https://olake.io/docs/install/kubernetes/): Deploy OLake Go on Kubernetes using Helm charts.
- [Streams v2: selected_streams.json and available_streams.json](https://olake.io/docs/install/docker-cli/#streams-config): Reference for selected_streams.json (editable stream settings) and available_streams.json (read-only discovered metadata) used by the OLake CLI.
- [Migrate from Streams v1 to Streams v2](https://olake.io/docs/core/configs/streams-migration/): Migration guide from Streams v1 (legacy single-file streams.json) to Streams v2 (selected_streams.json and available_streams.json).
- [Streams v1 (legacy streams.json) Reference](https://olake.io/docs/core/configs/streams-legacy/): Reference for Streams v1, the deprecated single-file streams.json format, kept for backwards compatibility and historical reference.
## Sources, Destinations & Core Concepts
- [PostgreSQL Source Connector](https://olake.io/docs/connectors/postgres/): Configuration guide for the OLake PostgreSQL source connector, covering full refresh, incremental, and CDC sync modes.
- [MongoDB Source Connector](https://olake.io/docs/connectors/mongodb/): Configuration guide for the OLake MongoDB source connector, covering full refresh, incremental, and CDC sync modes.
- [MySQL Source Connector](https://olake.io/docs/connectors/mysql/): Configuration guide for the OLake MySQL (and MariaDB) source connector, covering full refresh and CDC sync modes.
- [Oracle Source Connector](https://olake.io/docs/connectors/oracle/): Configuration guide for the OLake Oracle source connector, covering full refresh and incremental sync modes.
- [Kafka Source Connector](https://olake.io/docs/connectors/kafka/): Configuration guide for the OLake Kafka source connector, covering real-time message sync from topics.
- [Db2 Source Connector](https://olake.io/docs/connectors/db2/): Configuration guide for the OLake IBM Db2 LUW source connector, covering full refresh and incremental sync modes.
- [S3 Source Connector](https://olake.io/docs/connectors/s3/): Configuration guide for the OLake S3 source connector, covering CSV, JSON, and Parquet ingestion from AWS S3, MinIO, and LocalStack.
- [MSSQL Source Connector](https://olake.io/docs/connectors/mssql/): Configuration guide for the OLake MSSQL source connector, covering full refresh, incremental, and CDC sync modes.
- [Iceberg Destination and Catalogs (AWS Glue and more)](https://olake.io/docs/writers/iceberg/catalog/glue/): Configuring Apache Iceberg destinations across supported catalog integrations.
- [Iceberg Partitioning in OLake](https://olake.io/docs/writers/iceberg/partitioning/): How OLake handles metadata-driven Iceberg partitioning.
- [OLake Architecture](https://olake.io/docs/core/architecture/): Core architecture of the OLake platform.
- [Compatibility with Iceberg Catalogs](https://olake.io/docs/understanding/compatibility-catalogs/): Supported Iceberg catalog integrations including Glue, Hive Metastore, REST, and JDBC catalogs.
- [Compatibility with Query Engines](https://olake.io/docs/understanding/compatibility-engines/): Query engines that can read OLake-written Iceberg tables.
- [Schema Evolution in OLake](https://olake.io/docs/features/schema/): How OLake handles schema changes and data type changes without breaking downstream jobs.
- [OLake Use Cases](https://olake.io/docs/core/use-cases/): Common use cases for OLake across replication and lakehouse workloads.
- [OLake UI API](https://olake.io/docs/api/olake-ui-api/): API documentation for the OLake UI.
- [Release Notes](https://olake.io/docs/release/ingestion/overview/): Version history and release notes for OLake ingestion.
## Integration Guides (tested, step by step)
Each guide replicates one source into Apache Iceberg and queries it from one engine, written from a pipeline that was run end to end with OLake Go v1.1.0. Engine and catalog pages list which Iceberg catalogs and delete formats each engine reads.
- [Integrations: database to query engine, via Iceberg](https://olake.io/docs/integrations/): Tested guides that replicate Postgres, MySQL, MongoDB and more into Apache Iceberg and query them from Snowflake, BigQuery, Trino and DuckDB.
- [Amazon Athena and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/athena/): Replicate databases into Apache Iceberg tables in AWS Glue with OLake Go and query them in Amazon Athena: catalogs it reads, v2-only deletes, and guides.
- [BigQuery and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/bigquery/): Replicate databases to BigQuery as Apache Iceberg tables in the BigLake catalog with OLake Go: catalog support, delete formats, and nine tested guides.
- [ClickHouse and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/clickhouse/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in ClickHouse: DataLakeCatalog types, delete formats, six guides.
- [Databricks and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/databricks/): Replicate databases to Databricks as Apache Iceberg tables in Unity Catalog with OLake Go: catalog support, deletion vectors, and eight tested guides.
- [Dataproc and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/dataproc/): Read Apache Iceberg tables that OLake Go replicates into the BigLake catalog with Spark on Google Cloud Dataproc: catalogs, delete formats and limits.
- [Apache Doris and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/doris/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in Apache Doris: catalogs Doris reads and delete formats by release.
- [DuckDB and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/duckdb/): Query Apache Iceberg tables that OLake Go replicates from Postgres or MySQL in DuckDB: the REST catalogs DuckDB attaches, delete formats, tested guides.
- [Spark on Amazon EMR with Apache Iceberg and OLake Go](https://olake.io/docs/integrations/emr/): Read Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB with Spark on Amazon EMR: catalogs, delete formats, EMR releases.
- [Microsoft Fabric and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/fabric/): Read Apache Iceberg tables that OLake Go writes to Lakekeeper on Azure storage in Microsoft Fabric through OneLake shortcuts: what works and what fails.
- [Apache Iceberg with OLake Go: catalogs and guides](https://olake.io/docs/integrations/iceberg/): Replicate Postgres, MySQL, MongoDB and more to Apache Iceberg with OLake Go: the catalogs it writes to, delete formats, and the engines that read them.
- [Presto and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/presto/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in Presto: catalogs, Java versus C++ workers, and delete formats.
- [Amazon Redshift and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/redshift/): Replicate databases to Amazon Redshift as Apache Iceberg tables in AWS Glue with OLake Go: catalog support, delete formats, and nine tested guides.
- [Snowflake and Apache Iceberg: OLake Go guides](https://olake.io/docs/integrations/snowflake/): Replicate databases to Snowflake as Apache Iceberg tables with OLake Go: the catalogs Snowflake reads, the delete formats it takes, and nine tested guides.
- [Apache Spark and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/spark/): Read Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB with Apache Spark: catalog settings, delete formats, tested guides.
- [Starburst and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/starburst/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in Starburst Galaxy or Enterprise: catalogs and delete formats.
- [StarRocks and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/starrocks/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in StarRocks: catalogs StarRocks reads and its delete-format limits.
- [Trino and Apache Iceberg: OLake Go setup](https://olake.io/docs/integrations/trino/): Query Apache Iceberg tables that OLake Go replicates from Postgres, MySQL or MongoDB in Trino: every catalog Trino reads, delete formats, tested runs.
- [BigLake Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/biglake/): Replicate databases into Apache Iceberg tables in the BigLake REST catalog on Google Cloud with OLake Go: service account sign-in, readers, tested guides.
- [AWS Glue Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/glue/): Replicate databases into Apache Iceberg tables in the AWS Glue Data Catalog with OLake Go: authentication, the engines that read Glue, and tested guides.
- [Snowflake Horizon Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/horizon/): Replicate databases into Snowflake-managed Apache Iceberg tables through the Horizon Catalog REST API with OLake Go: authentication, readers, guides.
- [Lakekeeper Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/lakekeeper/): Replicate databases into Apache Iceberg tables in a self-hosted Lakekeeper REST catalog with OLake Go: authentication, readers and setup links.
- [Apache Polaris Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/polaris/): Replicate databases into Apache Iceberg tables in an Apache Polaris REST catalog with OLake Go: authentication, engines that read Polaris, tested guides.
- [Amazon S3 Tables Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/s3tables/): Replicate databases into Apache Iceberg tables in Amazon S3 Tables with OLake Go: SigV4 sign-in, the engines that read S3 Tables, and setup links.
- [MinIO AIStor Tables Iceberg catalog with OLake Go](https://olake.io/docs/integrations/catalogs/aistor/): Replicate databases into Apache Iceberg tables in MinIO AIStor Tables with OLake Go: SigV4 setup, the engines that read AIStor, and a tested guide.
- [Unity Catalog Iceberg tables with OLake Go](https://olake.io/docs/integrations/catalogs/unity/): Replicate databases into Unity Catalog managed Apache Iceberg tables with OLake Go: authentication options, the engines that read Unity, tested guides.
- [DB2 to BigQuery: Incremental Sync via Iceberg](https://olake.io/docs/integrations/db2-to-bigquery/): Replicate IBM DB2 LUW to BigQuery: OLake Go lands DB2 tables as Iceberg in a BigLake catalog on Cloud Storage and keeps them fresh on an UPDATED_AT cursor.
- [DB2 to ClickHouse: Incremental Sync via Iceberg](https://olake.io/docs/integrations/db2-to-clickhouse/): Replicate IBM DB2 LUW to ClickHouse: OLake Go writes DB2 tables as Iceberg into an Apache Polaris catalog on a cursor, and ClickHouse queries them.
- [DB2 to Redshift: Incremental Sync via Iceberg](https://olake.io/docs/integrations/db2-to-redshift/): Replicate IBM DB2 LUW to Amazon Redshift: OLake Go lands DB2 tables as Iceberg in the AWS Glue Data Catalog and keeps them fresh on an UPDATED_AT cursor.
- [DB2 to Snowflake: Incremental Sync via Iceberg](https://olake.io/docs/integrations/db2-to-snowflake/): Replicate IBM DB2 LUW to Snowflake: OLake Go lands DB2 tables as Iceberg in Snowflake's Horizon catalog and keeps them fresh on an UPDATED_AT cursor.
- [Kafka to BigQuery: Topics as Apache Iceberg](https://olake.io/docs/integrations/kafka-to-bigquery/): Stream Kafka topics to BigQuery as Iceberg tables: OLake Go upserts JSON messages into a BigLake catalog, and BigQuery, DuckDB and Spark read them.
- [Kafka to ClickHouse: Topics via Apache Iceberg](https://olake.io/docs/integrations/kafka-to-clickhouse/): Stream Kafka to ClickHouse as Iceberg tables: OLake Go upserts JSON messages into an Apache Polaris catalog, and ClickHouse, DuckDB and Trino query them.
- [Kafka to Databricks: Topics as Apache Iceberg](https://olake.io/docs/integrations/kafka-to-databricks/): Stream Kafka topics to Databricks as Apache Iceberg: OLake Go appends JSON messages into Unity Catalog, and Databricks, DuckDB and Spark query the tables.
- [Kafka to Apache Iceberg: Topics as Tables](https://olake.io/docs/integrations/kafka-to-iceberg/): Stream Kafka to Apache Iceberg without Kafka Connect: OLake Go upserts JSON messages into a Polaris catalog, and Spark, DuckDB and Trino query the tables.
- [Postgres to MinIO AIStor: CDC into Iceberg Tables](https://olake.io/docs/integrations/postgres-to-aistor/): Replicate Postgres to MinIO AIStor with WAL CDC: OLake Go writes Iceberg tables into AIStor Tables, and Spark, DuckDB and Trino query them.
- [Kafka to Redshift: Topics as Apache Iceberg](https://olake.io/docs/integrations/kafka-to-redshift/): Stream Kafka topics to Amazon Redshift as Iceberg tables: OLake Go upserts JSON messages into AWS Glue, and Redshift, Athena and DuckDB query them.
- [Kafka to Snowflake: Topics as Apache Iceberg](https://olake.io/docs/integrations/kafka-to-snowflake/): Stream Kafka topics to Snowflake as Iceberg tables: OLake Go upserts JSON messages into the Horizon catalog, and Snowflake, DuckDB and Spark query them.
- [MariaDB to BigQuery: CDC via Apache Iceberg](https://olake.io/docs/integrations/mariadb-to-bigquery/): Replicate MariaDB to BigQuery with binlog CDC: OLake Go writes Iceberg tables to a BigLake catalog on Cloud Storage, read by BigQuery, DuckDB and Spark.
- [MariaDB to Databricks: CDC via Apache Iceberg](https://olake.io/docs/integrations/mariadb-to-databricks/): Replicate MariaDB to Databricks with binlog CDC: OLake Go lands Iceberg tables in Unity Catalog that Databricks, DuckDB and Spark query directly.
- [MariaDB to Redshift: CDC via Apache Iceberg](https://olake.io/docs/integrations/mariadb-to-redshift/): Replicate MariaDB to Amazon Redshift with binlog CDC: OLake Go writes Iceberg tables to the AWS Glue Data Catalog for Redshift, Athena and DuckDB to query.
- [MariaDB to Snowflake: CDC via Apache Iceberg](https://olake.io/docs/integrations/mariadb-to-snowflake/): Replicate MariaDB to Snowflake with binlog CDC: OLake Go writes Iceberg tables to Snowflake's Horizon catalog, and Snowflake, DuckDB and Spark query them.
- [MongoDB to BigQuery: CDC via Apache Iceberg](https://olake.io/docs/integrations/mongodb-to-bigquery/): Replicate MongoDB to BigQuery with change stream CDC: OLake Go writes Iceberg tables to a BigLake catalog on Cloud Storage for BigQuery, DuckDB and Spark.
- [MongoDB to ClickHouse: CDC via Apache Iceberg](https://olake.io/docs/integrations/mongodb-to-clickhouse/): Replicate MongoDB to ClickHouse with change stream CDC: OLake Go writes Iceberg tables to a Polaris catalog that ClickHouse, DuckDB and Trino query.
- [MongoDB to Databricks: CDC via Apache Iceberg](https://olake.io/docs/integrations/mongodb-to-databricks/): Replicate MongoDB to Databricks with change streams: OLake Go writes Iceberg tables into Unity Catalog for Databricks, DuckDB and Spark to query.
- [MongoDB to Apache Iceberg: CDC without Kafka](https://olake.io/docs/integrations/mongodb-to-iceberg/): Replicate MongoDB to Apache Iceberg without Kafka: OLake Go reads change streams into Iceberg tables in a Polaris catalog for Spark, DuckDB and Trino.
- [MongoDB to Redshift: CDC via Apache Iceberg](https://olake.io/docs/integrations/mongodb-to-redshift/): Replicate MongoDB to Amazon Redshift with change stream CDC: OLake Go writes Iceberg tab