You'll want it buttoning more than just on the one side; I'd say sewn on one side (which was no doubt the plan) and either buttons on all three other sides, or velcro, snaps or something. Otherwise that front cover panel will end up moving oddly whenever you twist or bend certain ways.
Heck, you could probably get away with covering the front of the vest with velcro, then covering the back of the cover panel with the other side of the velcro. Mount the armor to the cover panel, put the vest on and velcro the cover panel in place. Quick and easy like.