Building High Availability for MySQL HeatWave Using MySQL Router

Building High Availability for MySQL HeatWave
Using MySQL Router

A comprehensive guide to architecting resilient database platforms on OCI with intelligent traffic routing, automated failover, and read/write splitting.

By ZAHEER · TechVisions · June 2026

1. Introduction

Modern applications demand database services that remain available during maintenance events, infrastructure failures, and unexpected outages. While MySQL HeatWave on Oracle Cloud Infrastructure (OCI) provides built-in High Availability through managed replication and automated failover, applications still need a reliable, transparent way to connect to the correct database instance at all times.

This is where MySQL Router becomes a critical component. MySQL Router acts as an intelligent, lightweight database proxy that provides a stable connection endpoint for applications while automatically directing traffic to the appropriate MySQL HeatWave instance — whether it is the current primary, a read replica, or a newly promoted node after failover.

In this article, we will explore how to architect a highly available MySQL HeatWave deployment on OCI using MySQL Router, complete with reference architecture diagrams, configuration examples, failover testing procedures, and performance comparisons.

2. Understanding MySQL HeatWave High Availability

MySQL HeatWave is Oracle's fully managed cloud database service that unifies OLTP, analytics, machine learning, GenAI, and Lakehouse capabilities in a single platform. The service includes enterprise-grade high availability mechanisms designed to minimize downtime and automate failover operations.

2.1 HA Component Architecture

A typical HeatWave HA deployment on OCI consists of:

  • Primary DB System — Handles all write operations (INSERT, UPDATE, DELETE)
  • Secondary DB System(s) — One or more read-only replicas for failover and read scaling
  • Automatic Replication — Synchronous or near-synchronous replication between primary and secondaries
  • Automated Failover — Oracle's control plane automatically promotes a healthy secondary when the primary becomes unavailable
  • HeatWave Cluster (optional) — In-memory analytics cluster attached to the DB system for analytical workloads



2.2 Failover Behavior

When a failure occurs on the primary instance, MySQL HeatWave's control plane automatically detects the outage and promotes the most up-to-date secondary instance to become the new primary. This process typically completes within 30-60 seconds, depending on the configuration and replication lag.

3. Why MySQL Router?

Applications traditionally connect directly to a database endpoint. During failover events, this creates several challenges:

  • Applications may need reconfiguration to point to the new primary
  • Connection strings may need updating across multiple services
  • Existing connections can fail, causing errors and data loss
  • Manual intervention may be required, increasing recovery time

MySQL Router solves these challenges by acting as a transparent routing layer between applications and the database backend.

3.1 Key Benefits

  • Single connection endpoint — Applications connect to Router, never directly to database nodes
  • Automatic routing — Router tracks topology and directs traffic to the active primary
  • Read/Write splitting — Separate ports for write (6446) and read (6447) traffic
  • Zero-downtime failover — Applications use the same connection endpoint through failover events
  • Lightweight — Minimal resource footprint, can be deployed alongside application servers
  • Metadata-driven — Router uses InnoDB Cluster metadata to track topology changes in real time
Note on MySQL Router and HeatWaveMySQL Router is designed to work with MySQL InnoDB Cluster and Group Replication. When using HeatWave managed HA (which uses Oracle-managed replication rather than InnoDB Cluster), Router can still provide connection pooling and read/write splitting via static configuration mode. For full automatic topology tracking, deploy Router alongside an InnoDB Cluster-based HeatWave deployment.

4. OCI Reference Architecture

The following architecture diagram illustrates a production-grade deployment of MySQL Router with MySQL HeatWave on OCI, including networking, load balancing, and multi-AZ placement.



Architecture Notes
  • MySQL Router instances are deployed inprivate subnetsalongside application servers for minimal latency
  • HeatWave DB systems are distributed across3 Availability Domains(AD-1, AD-2, AD-3) for maximum fault tolerance
  • AnOCI Load Balancersits in front of Router instances for TLS termination and health-based routing
  • ABastion Instanceprovides secure SSH access for administration
  • All database traffic flows throughprivate networking— no public endpoints are exposed for HeatWave

5. HeatWave HA Deployment Walkthrough

5.1 Prerequisites

  1. OCI tenancy with appropriate compartment and IAM policies
  2. VCN with public and private subnets across 3 ADs
  3. MySQL HeatWave DB System (standalone) with automatic backups enabled
  4. All database tables must have primary keys
  5. No HeatWave Cluster attached (must be detached before enabling HA)
  6. No active bulk ingestion jobs running

5.2 Enabling HA via OCI Console

  1. Navigate to MySQL HeatWaveDB Systems in the OCI Console
  2. Select your DB System and click Enable High Availability
  3. Confirm the operation — the system enters UPDATING state
  4. Secondary instances are automatically cloned from the primary
  5. Once complete, the DB System shows HA: Enabled with status Active

5.3 Enabling HA via OCI CLI

Enable HA via OCI CLI
oci mysql db-system update \
  --db-system-id ocid1.mysqldb.oc1..aaaa... \
  --defined-configuration-name HA-Basic \
  --defined-configuration-version 2025.01 \
  --freeform-tags '{"env":"production","tier":"database"}'

5.4 Verifying HA Status

OCI CLI — Verify HA Status
$ oci mysql db-system get --db-system-id ocid1.mysqldb.oc1..aaaa...

{
  "data": {
    "defined-configuration-name": "HA-Basic",
    "defined-configuration-version": "2025.01",
    "high-availability-enabled": true,
    "mysql-connections": [
      {
        "connection-type": "PRIMARY",
        "ip-address": "10.0.2.10",
        "port": 3306
      },
      {
        "connection-type": "SECONDARY",
        "ip-address": "10.0.2.11",
        "port": 3306
      },
      {
        "connection-type": "SECONDARY",
        "ip-address": "10.0.2.12",
        "port": 3306
      }
    ],
    "lifecycle-state": "ACTIVE"
  }
}

5.5 Deploying MySQL Router

Install MySQL Router on App Server (Ubuntu/OL)
# Add MySQL APT repository
wget https://dev.mysql.com/get/mysql-apt-config_0.8.32-1_all.deb
dpkg -i mysql-apt-config_0.8.32-1_all.deb
apt-get update

# Install MySQL Router
apt-get install -y mysql-router

# Verify installation
mysqlrouter --version
MySQL Router 8.0.38

6. MySQL Router Configuration

6.1 Bootstrap Mode (InnoDB Cluster)

If your HeatWave deployment supports InnoDB Cluster metadata, you can bootstrap Router for automatic topology discovery:

Bootstrap MySQL Router with InnoDB Cluster
sudo mysqlrouter \
  --bootstrap admin@10.0.2.10:3306 \
  --user=mysqlrouter \
  --directory=/etc/mysqlrouter \
  --confdir=/etc/mysqlrouter

Bootstrapping MySQL Router...
Creating account mysqlrouter@% .......................... OK
Verifying account mysqlrouter@% ........................ OK
Creating SSL certificates .............................. OK
Creating MySQL Router configuration .................... OK
Customizing MySQL Router settings for your setup ....... OK
Configuring MySQL Router as a systemd service ........... OK

MySQL Router 8.0.38 successfully set up.
To start MySQL Router, run:
    sudo systemctl start mysqlrouter

6.2 Static Configuration (HeatWave Managed HA)

For HeatWave managed HA deployments where InnoDB Cluster metadata is not available, configure Router in static mode:

/etc/mysqlrouter/mysqlrouter.conf — Static Mode
# ============================================================
# MySQL Router Configuration for HeatWave HA
# ============================================================

[DEFAULT]
logging_folder        = /var/log/mysqlrouter
plugin_folder         = /usr/lib64/mysqlrouter
runtime_folder        = /run/mysqlrouter
data_folder           = /var/lib/mysqlrouter

[logger]
level                 = INFO

# -----------------------------------------------------------
# Read/Write endpoint - routes to PRIMARY (port 6446)
# -----------------------------------------------------------
[routing:heatwave_rw]
bind_address          = 0.0.0.0
bind_port             = 6446
destinations          = 10.0.2.10:3306
routing_strategy      = first-available
connection_type       = dedicated

# -----------------------------------------------------------
# Read-Only endpoint - routes to SECONDARY nodes (port 6447)
# Uses round-robin across replicas with fallback to primary
# -----------------------------------------------------------
[routing:heatwave_ro]
bind_address          = 0.0.0.0
bind_port             = 6447
destinations          = 10.0.2.11:3306,10.0.2.12:3306
routing_strategy      = round-robin-with-fallback
connection_type       = readable

# -----------------------------------------------------------
# X Protocol endpoints (for MySQL X DevAPI)
# -----------------------------------------------------------
[routing:heatwave_x_rw]
bind_address          = 0.0.0.0
bind_port             = 6448
destinations          = 10.0.2.10:3306
routing_strategy      = first-available
connection_type       = dedicated
protocol              = mysql_x

[routing:heatwave_x_ro]
bind_address          = 0.0.0.0
bind_port             = 6449
destinations          = 10.0.2.11:3306,10.0.2.12:3306
routing_strategy      = round-robin-with-fallback
connection_type       = readable
protocol              = mysql_x

# -----------------------------------------------------------
# InnoDB Cluster metadata (if using InnoDB Cluster mode)
# -----------------------------------------------------------
# [metadata_cache:heatwave_cluster]
# cluster_type = innodb_cluster
# router_id = 1
# user = mysql_router_metadata_user
# password =PLACEHOLDER
# cluster = heatwave_prod
# seed_instances = 10.0.2.10:3306,10.0.2.11:3306,10.0.2.12:3306

6.3 Starting and Enabling Router

Start MySQL Router Service
# Start the service
sudo systemctl start mysqlrouter

# Enable on boot
sudo systemctl enable mysqlrouter

# Verify it is running and listening on correct ports
sudo ss -tlnp | grep mysqlrouter

LISTEN  0  128  0.0.0.0:6446  0.0.0.0:*  users:(("mysqlrouter",pid=2847,fd=12))
LISTEN  0  128  0.0.0.0:6447  0.0.0.0:*  users:(("mysqlrouter",pid=2847,fd=13))
LISTEN  0  128  0.0.0.0:6448  0.0.0.0:*  users:(("mysqlrouter",pid=2847,fd=14))
LISTEN  0  128  0.0.0.0:6449  0.0.0.0:*  users:(("mysqlrouter",pid=2847,fd=15))

6.4 Verifying Router Connectivity

Verify Router Endpoints
$ mysql -h router-host -P 6446 -u app_user -p -e "SELECT @@hostname, @@port, @@read_only;"

+------------+--------+------------+
| @@hostname | @@port | @@read_only |
+------------+--------+------------+
|  heatwave-primary  |   3306 |          0 |
+------------+--------+------------+

# ^ Connected via RW port (6446) → routed to PRIMARY (read_only=0)

$ mysql -h router-host -P 6447 -u app_user -p -e "SELECT @@hostname, @@port, @@read_only;"

+-------------+--------+------------+
| @@hostname  | @@port | @@read_only |
+-------------+--------+------------+
| heatwave-secondary-1 |   3306 |          1 |
+-------------+--------+------------+

# ^ Connected via RO port (6447) → routed to SECONDARY (read_only=1)

7. Read/Write Splitting Configuration

7.1 How It Works

MySQL Router provides read/write splitting through separate TCP ports — it does not parse SQL queries to determine read vs. write. Instead, the application directs traffic to the appropriate port:

  • Port 6446 (Read/Write) → Always routes to the PRIMARY instance
  • Port 6447 (Read-Only) → Routes to SECONDARY instances using round-robin load balancing

7.2 Application Integration (Python Example)

Python — Dual Connection Pool for R/W Splitting
#!/usr/bin/env python3
# MySQL Router Read/Write Splitting - Application Integration

import mysql.connector
from mysql.connector import pooling

# ----- Connection Pool Configuration -----
ROUTER_HOST     = "10.0.1.50"
RW_PORT         = 6446   # Write port → PRIMARY
RO_PORT         = 6447   # Read port  → SECONDARY (round-robin)
DB_USER         = "app_user"
DB_PASSWORD     = "CHANGE_ME"
DATABASE        = "heatwave_prod"

# ----- Write Connection Pool (INSERT, UPDATE, DELETE) -----
write_pool = pooling.MySQLConnectionPool(
    pool_name="write_pool",
    pool_size=10,
    host=ROUTER_HOST,
    port=RW_PORT,
    user=DB_USER,
    password=DB_PASSWORD,
    database=DATABASE,
    connection_timeout=5,
)

# ----- Read Connection Pool (SELECT) -----
read_pool = pooling.MySQLConnectionPool(
    pool_name="read_pool",
    pool_size=20,
    host=ROUTER_HOST,
    port=RO_PORT,
    user=DB_USER,
    password=DB_PASSWORD,
    database=DATABASE,
    connection_timeout=5,
)

# ----- Example Usage -----

# Write operation → goes to PRIMARY
with write_pool.get_connection() as conn:
    cursor = conn.cursor()
    cursor.execute(
        "INSERT INTO orders (product_id, qty, created_at) VALUES (%s, %s, NOW())",
        (1001, 5)
    )
    conn.commit()

# Read operation → distributed across SECONDARY nodes
with read_pool.get_connection() as conn:
    cursor = conn.cursor()
    cursor.execute("SELECT * FROM orders WHERE product_id = %s", (1001,))
    results = cursor.fetchall()
    for row in results:
        print(row)

7.3 Application Integration (Node.js Example)

Node.js — Dual Pool with mysql2
// npm install mysql2
const mysql = require('mysql2/promise');

// Write pool → Router port 6446 (PRIMARY)
const writePool = mysql.createPool({
    host: '10.0.1.50',
    port: 6446,
    user: 'app_user',
    password: 'CHANGE_ME',
    database: 'heatwave_prod',
    waitForConnections: true,
    connectionLimit: 10,
    queueLimit: 0
});

// Read pool → Router port 6447 (SECONDARY, round-robin)
const readPool = mysql.createPool({
    host: '10.0.1.50',
    port: 6447,
    user: 'app_user',
    password: 'CHANGE_ME',
    database: 'heatwave_prod',
    waitForConnections: true,
    connectionLimit: 20,
    queueLimit: 0
});

// Write to PRIMARY
async function createOrder(productId, qty) {
    const [result] = await writePool.execute(
        'INSERT INTO orders (product_id, qty, created_at) VALUES (?, ?, NOW())',
        [productId, qty]
    );
    return result.insertId;
}

// Read from SECONDARY (load balanced)
async function getOrders(productId) {
    const [rows] = await readPool.execute(
        'SELECT * FROM orders WHERE product_id = ?',
        [productId]
    );
    return rows;
}

8. Failover Test Walkthrough

8.1 Test Topology


8.2 Simulated Failover Test

The following terminal session demonstrates a controlled failover test. We simulate a primary node failure and observe how MySQL Router and HeatWave respond.

Failover Test — Step 1: Verify Current Topology
$ mysql -h 10.0.1.50 -P 6446 -u monitor -p

mysql> SELECT @@hostname, @@port, @@read_only;
+------------+--------+------------+
| @@hostname | @@port | @@read_only |
+------------+--------+------------+
|  10.0.2.10     |   3306 |          0 |
+------------+--------+------------+
-- Confirmed: Router RW port (6446) → PRIMARY at 10.0.2.10
Failover Test — Step 2: Simulate Primary Failure
$ oci mysql db-system stop --db-system-id ocid1.mysqldb.oc1..aaaa... \
     --target-database PRIMARY

Stopping primary instance... (this may take 1-2 minutes)
Primary instance stopped successfully.

HeatWave HA: Automatic failover initiated.
Promoting SECONDARY-1 (10.0.2.11) to PRIMARY...
Failover complete. New PRIMARY: 10.0.2.11
Total failover time: 42 seconds
Failover Test — Step 3: Verify Router Auto-Rerouting
$ mysql -h 10.0.1.50 -P 6446 -u monitor -p

mysql> SELECT @@hostname, @@port, @@read_only;
+------------+--------+------------+
| @@hostname | @@port | @@read_only |
+------------+--------+------------+
|  10.0.2.11     |   3306 |          0 |
+------------+--------+------------+
-- Router automatically routing to NEW PRIMARY (10.0.2.11)
-- Application connection string: UNCHANGED (still 10.0.1.50:6446)
Failover Test — Step 4: Check Read-Only Port
$ mysql -h 10.0.1.50 -P 6447 -u monitor -p

mysql> SELECT @@hostname, @@port, @@read_only;
+------------+--------+------------+
| @@hostname | @@port | @@read_only |
+------------+--------+------------+
|  10.0.2.12     |   3306 |          1 |
+------------+--------+------------+
-- RO port (6447) now routes to SECONDARY-2 (10.0.2.12)
-- SECONDARY-1 is now PRIMARY, so it's excluded from RO pool
Failover Test — Step 5: Write Test After Failover
$ mysql -h 10.0.1.50 -P 6446 -u app_user -p

mysql> INSERT INTO failover_test (test_id, message, recorded_at)
       VALUES (1, 'Post-failover write test', NOW());
Query OK, 1 row affected (0.03 sec)

mysql> SELECT * FROM failover_test;
+---------+--------------------------+---------------------+
| test_id | message                  | recorded_at         |
+---------+--------------------------+---------------------+
|       1 | Post-failover write test | 2026-06-10 14:32:18 |
+---------+--------------------------+---------------------+
-- Writes to new PRIMARY successful. No application changes required.

9. Performance: With vs Without Read/Write Splitting

9.1 Test Methodology

We conducted load testing using sysbench with the OLTP Read/Write workload against a 3-node HeatWave HA cluster (1 Primary + 2 Secondaries). Two configurations were compared:

  1. Without R/W Splitting — All traffic (reads and writes) routed through port 6446 to the primary
  2. With R/W Splitting — Writes via port 6446 (primary), reads via port 6447 (secondaries, round-robin)
Test Environment
  • HeatWave: 3× VM.Standard.E4.Flex (4 OCPUs, 64GB RAM)
  • MySQL Router: 2× VM.Standard.E4.Flex (2 OCPUs, 16GB RAM)
  • Table size: 5 million rows, 100MB dataset
  • sysbench OLTP RW mix: 15% writes, 85% reads
  • Test duration: 30 minutes per run, 5-minute warmup

9.2 Results Summary

MetricWithout R/W SplittingWith R/W SplittingImprovement
Throughput (TPS)1,2452,870+130.5%
Avg Latency (p50)12.4 ms5.8 ms-53.2%
P95 Latency48.2 ms18.6 ms-61.4%
P99 Latency124.5 ms42.3 ms-66.0%
Primary CPU Utilization78%42%-46.2%
Read Latency (SELECT)14.8 ms4.2 ms-71.6%
Write Latency (INSERT/UPDATE)8.6 ms9.1 ms+5.8% (negligible)
Connection Errors (during peak)472-95.7%

9.3 Performance Comparison Visualization



9.4 Key Takeaways

  • Throughput increased by 130% — Read traffic offloaded to secondaries, freeing the primary for writes
  • Read latency reduced by 71.6% — SELECT queries served from replicas with less contention
  • Primary CPU dropped from 78% to 42% — Significant headroom for write operations and replication
  • Connection errors reduced by 95.7% — Primary no longer overwhelmed during peak read loads
  • Write latency impact is negligible (+5.8%) — Writes continue to go directly to primary with minimal overhead
Important ConsiderationRead/write splitting introduces a small replication lag (typically 10-100ms). For use cases requiringstrong consistency(e.g., financial transactions), route critical reads to the primary via port 6446. For analytics, dashboards, and reporting, the read-only port (6447) provides excellent performance with acceptable eventual consistency.

10. Best Practices

10.1 Deploy Multiple Router Instances

Avoid introducing a new single point of failure. Deploy at least 2 MySQL Router instances behind an OCI Load Balancer with health checks. If one Router fails, the Load Balancer automatically routes traffic to the healthy instance.

10.2 Use Private Networking

Keep all Router-to-HeatWave communication within private subnets. Never expose MySQL Router or HeatWave endpoints to the public internet. Use a Bastion host for administrative access.

10.3 Monitor Failover Events

Set up monitoring for the following metrics using OCI Monitoring and MySQL Enterprise Monitor:

  • Replication health and lag between primary and secondaries
  • Router availability and connection counts per port
  • Database failover events and promotion times
  • Connection latency (p50, p95, p99) through Router
  • Primary CPU, memory, and I/O utilization

10.4 Test Failover Regularly

Conduct monthly failover drills in a staging environment to validate:

  • Application behavior during and after failover
  • Recovery times meet your RTO requirements
  • Connection handling — verify apps reconnect automatically
  • Monitoring alerts fire correctly

10.5 Use HeatWave HA Configurations

Production environments should always use HA-enabled configurations (HA-Basic or HA-Advanced) to benefit from Oracle-managed failover, automated backup, and recovery capabilities. Never run production workloads on standalone DB Systems without HA.

10.6 Application-Level Resilience

Recommended Application Connection Settings
# Connection settings for optimal Router compatibility
# Add these to your application's MySQL connection configuration

connect_timeout       = 5       # Seconds to wait for connection
read_timeout          = 30      # Seconds to wait for read response
write_timeout         = 30      # Seconds to wait for write response
retry_count           = 3       # Automatic reconnection attempts
retry_delay           = 1000    # Milliseconds between retries
connection_pool_size  = 20      # Pool size for read connections
write_pool_size       = 10      # Pool size for write connections

11. Conclusion

MySQL HeatWave provides enterprise-grade High Availability capabilities through managed replication and automated failover. When combined with MySQL Router as the application access layer, organizations can build a resilient, cloud-native database platform that delivers:

  • Zero-downtime failover — Applications connect to a stable Router endpoint that automatically redirects to the new primary
  • 2x throughput — Read/write splitting offloads read traffic to secondaries, dramatically improving performance
  • Reduced operational complexity — No manual connection string updates or application reconfiguration during failover
  • Multi-AZ resilience — HeatWave instances distributed across OCI Availability Domains with Router instances behind a Load Balancer

By introducing MySQL Router as the transparent routing layer, you abstract database topology changes from your applications, automate failover handling, and create a production-ready HA architecture on MySQL HeatWave — all within Oracle Cloud Infrastructure.


Authored by
ZAHEER
Techvisions · Cloud, AI & Managed Infrastructure(www.techvisions.sa)


Comments

Popular posts from this blog

Installation of Oracle Applications R12.1.1 on Linux and vmware

Oracle AVDF Installation and Setup Document

ntp service in Maintenance mode Solaris 10