TokuDB Slave of PXC Master: Issues Encountered

For these findings, it is important to note that TokuDB was configured on a slave that was replicating from a PXC master. We could have wanted to test Writes on TokuDB, but unfortunately most of our databases are setup as PXC, and TokuDB is not yet supported by PXC servers. 

In the near future, i will try to test TokuDB as a single Percona server against a single Percona server with InnoDB, or even as a master of a slave with InnoDB. This way, we get the claimed enhanced performance from the Master with TokuDB by writing to it only, and ruse its slave with InnoDB for reads. Maybe this might produce better overall performance than an all InnoDB master-slave replication.

1. replication issues on MySQL 5.6:

Slaves are started with binlog enabled and slave updates being logged. The TokuDB engine has chosen binlog not mmap as the transaction coordinator. This is causing the warning message shown below that relay-log is not being used:

2016-02-25 16:42:04 9604 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=db-kdb-slave-6-relay-bin' to avoid this problem.
2016-02-25 16:42:05 9604 [Warning] Recovery from master pos 552554502 and file mysqld-bin.001163. Previous relay log pos and relay log file had been set to 552554714, ./db-kdb-slave-6-relay-bin.002933 respectively.
2016-02-25 16:42:05 9604 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.

With this log message, i think it would be better to try and disable log-bin on the slave and just run relay-log.

 Even though we do not need log-bin on the slave, we are using GTID which requires --log-slave-updates to be enabled on the slave, which also requires the log-bin to be enabled in the replication slave thread.

The problem here is this; The server can use either mmap-based transaction coordinator for 2PC or a binary log. 2PC always uses binary log, if binary logging is enabled. To possibly avoid this issue, one way is to disable GTID so that we do not need to use log-bin on the slave. But i am sure this would not be a solution that is welcome.

2. High CPU usage

During tests with TokuDB used on slaves, the CPU usage was super duper high. If the load is minimal, the servers were able to run just fine with normal CPU usage. But as soon as the load increases, this caused CPU usage to rise significantly.

With TokuDB, It seems like CPU will always be higher for reads because TokuDB data needs to be decompressed to be used. Also, since the slave is processing activities from the master for replication, then it has to do compression for the insert/update/delete activity, also contributing to high CPU usage. So basically, though compression and decompression saves IO, it is at the expense of CPU, and hence CPU will always be higher with TokuDB.

In the above image, the third column on slave-8 with the red angry face is for CPU.

You can also see in cacti CPU usage on the same slave-8 below:

I tried three different tests where tokudb_cache_size was set to 50%, and in the second test were it was set to 80% and tokudb_block_size reduced to 256KB. The last test i did involved commenting all innodb variables in my.cnf so the server can just use their default values to avoid any impact they may be having on database with only TokuDB tables. The reason for these tests was to see if the tokudb_cache_size and tokudb_block_size variables might be playing a part in High CPU usage. However, all three tests continued to show very high CPU usage.

Conclusion

TokuDB does not perform well, when used as a slave, and that slave is started with binlog enabled, as it will choose binlog and not mmap as the transaction coordinator. TokuDB also has high CPU usage. To understand how binary log commit works in TokuDB engine, this github blog.

The TokuDB project has not grown much since Percona bought it off from Tokutek. Its seems like they do not have the expertise, time or even plan to take it to the next level. One evidence is their lack of effort to include it in their biggest product, PXC, and also a lack of effort in creating an open source backup mechanism or tool for performing backups. Backups still depend on Tokutek's hot backup, which is only provided with the enterprise edition provided by Tokutek.