Occasionally, a business will obtain products or services from a vendor who is also a customer.  Rather than exchanging checks, a business can simply apply balances owed to balances owing, and vice versa.  QuickBooks does not allow a single transaction to use more than one A/R or A/P account, which makes this common business practice slightly cumbersome to record.  But only slightly.

The entry you are ultimately trying to perform is this:

Accounts Payable            $XX.XX

       Accounts Receivable              $XX.XX

There are a few ways to skin this cat, but my personal preference is this:

1) Determine the amount that is to be transferred.

2) [Receive Payment] in that amount on your outstanding invoice, depositing the "funds" into your Petty Cash "bank" account

3) [Pay Bills] in that amount on the bill you owe, using that same Petty Cash "bank" account.

Be sure to clearly note these payments, and you now have successfully transferred the debt balance appropriately.