5.7 mtr failed with wrong errno 1399: 'XAER_RMFAIL: The command cannot be executed when global transaction is in the PREPARED state', instead of 1397...

George, your findings look correct. The 5.7 TokuDB behavior seems to be inline with 5.7.7+ server changes. But please consider reporting the InnoDB scenario on bugs.mysql.com. It looks like a data loss bug to me, and quite serious one for a XA user.

Deeper testing on 5.6:
If we attempt the drop t1 at any point where there is an active XA transaction with a lock on t1, the drop table waits for the transaction to either commit or rollback, correct behavior IMHO. If the session that initially created and prepared the XA transaction terminates before commit/rollback, the entire transaction and all locks are discarded, thus allowing the table to be safely dropped in another session once the locks are relinquished.

Deeper testing on 5.7:
Badly broken exactly as described above with InnoDB. When the session that created and prepared the XA transaction terminates, it leaves the XA transaction behind, but releases the locks, allowing another session to drop t1 and then commit the XA transaction with no errors or warnings.

Registered PS bug https://bugs.launchpad.net/percona-server/+bug/1533364

Registered upstream https://bugs.mysql.com/bug.php?id=79940

For now, fixing the expected errors and adjusting the test case to expect the current behavior while awaiting an upstream fix, then will revisit the test.

For both 5.6 and 5.7:
Added new TOKUDB_DEBUG_XA (1<<15) debug flag for system variable tokudb_debug to debug log XA function calls only
Instrumented xa specific handlerton functions with enter/exit logging.
Converted existing TOKUDB_DEBUG_TXN logging within the XA functions to TOKUDB_DEBUG_XA.

Additionally for 5.7:

• tokudb.bugs.xa

• changed expected error from ER_XAER_NOTA to ER_XAER_RMFAIL and re-recorded

• tokudb.tokudb_support_xa

• changed expected error from ER_XAER_NOTA to ER_XAER_RMFAIL and re-recorded

• fixed test sequence due to changes in 5.7.7 that leaves XA transactions open if a session terminates, preventing a 'drop table' from succeeding correctly. See https://bugs.launchpad.net/percona-server/+bug/1533364 and https://bugs.mysql.com/bug.php?id=79940

• re-recorded results

5.6 merged in pull request https://github.com/percona/percona-server/pull/311
5.7 merged in pull request https://github.com/percona/percona-server/pull/312