I only withdraw dividends / distributions earned in my TFSA. I have never withdrawn Capital, only the income my Capital produces.
I gave the example of withdrawing of all funds in TFSA to exaggerate my point.
Goal, reduce taxable income. Withdraw income from TFSA and replace these withdrawals (now added to next year's contribution room) the following January with Capital from Margin account. Invest the new contribution funds and my income (tax free) in TFSA gradually moves higher and the taxable income in my Margin account reduces.