Keep a record of the most recently drawn character and the foreground and background colours. In bitblt_text functions, compare values to this cache and don't re-draw the characters if they haven't changed.
In particular, when scrolling the display (which is where most time is spent within the vt driver) there can be a significant performance improvement when most lines are less than the width of the terminal, since this avoids re-drawing blanks on top of blanks.
Note that "re-drawing" here includes writing to the VGA text mode buffer; on virtualized systems this can be extremely slow (since it triggers a glyph being rendered onto a 640x480 screen).