Happens in tests:
tokudb.bugs.xa : At line 72: query 'xa commit 'testb',0x2030405060,11'
tokudb.tokudb_support_xa : At line 116: query 'xa commit 'testb',0x2030405060,11'
George, your findings look correct. The 5.7 TokuDB behavior seems to be inline with 5.7.7+ server changes. But please consider reporting the InnoDB scenario on bugs.mysql.com. It looks like a data loss bug to me, and quite serious one for a XA user.
Deeper testing on 5.6:
If we attempt the drop t1 at any point where there is an active XA transaction with a lock on t1, the drop table waits for the transaction to either commit or rollback, correct behavior IMHO. If the session that initially created and prepared the XA transaction terminates before commit/rollback, the entire transaction and all locks are discarded, thus allowing the table to be safely dropped in another session once the locks are relinquished.
Deeper testing on 5.7:
Badly broken exactly as described above with InnoDB. When the session that created and prepared the XA transaction terminates, it leaves the XA transaction behind, but releases the locks, allowing another session to drop t1 and then commit the XA transaction with no errors or warnings.
Registered PS bug https://bugs.launchpad.net/percona-server/+bug/1533364
Registered upstream https://bugs.mysql.com/bug.php?id=79940
For now, fixing the expected errors and adjusting the test case to expect the current behavior while awaiting an upstream fix, then will revisit the test.
For both 5.6 and 5.7:
Added new TOKUDB_DEBUG_XA (1<<15) debug flag for system variable tokudb_debug to debug log XA function calls only
Instrumented xa specific handlerton functions with enter/exit logging.
Converted existing TOKUDB_DEBUG_TXN logging within the XA functions to TOKUDB_DEBUG_XA.
Additionally for 5.7:
tokudb.bugs.xa
changed expected error from ER_XAER_NOTA to ER_XAER_RMFAIL and re-recorded
tokudb.tokudb_support_xa
changed expected error from ER_XAER_NOTA to ER_XAER_RMFAIL and re-recorded
fixed test sequence due to changes in 5.7.7 that leaves XA transactions open if a session terminates, preventing a 'drop table' from succeeding correctly. See https://bugs.launchpad.net/percona-server/+bug/1533364 and https://bugs.mysql.com/bug.php?id=79940
re-recorded results
5.6 merged in pull request https://github.com/percona/percona-server/pull/311
5.7 merged in pull request https://github.com/percona/percona-server/pull/312