Here is the item again:
His arms full of expensive science fiction paperbacks, (A) Cary remembered (B) that next semester's tuition was due in two weeks. (C) Making a trip to the library instead of the bookstore a better financial decision.

You correctly realized that a comma was required after weeks. Making a trip to the library ... is a participle phrase fragment, and when you connect such a phrase to the end of a main clause, you will often need a comma.

To fix this problem, you chose to do this:

His arms full of expensive science fiction paperbacks, Cary remembered that next semester's tuition was due in two weeks, making a trip to the library instead of the bookstore a better financial decision.

You have correctly employed Punctuation Rule 3:

Main Clause + , + Participle Phrase.

We will hum that tune all day!

Go to the next item.