How do I know if my fin is too big?

You will have too much lift.

You will start to lose control very quickly, the nose of the board will start to lift and all of the board will follow as soon as a gust of wind hits the sail.

You can sail overpowered and with a little bigger fin but you have to be a very powerful and  an expert sailor. When this happens the common mistake is to open the sail. You have to be strong and keep your sail closed, keep sheeted in and try to push the board down to gain more control. If you open the sail you will totally lose control.