I created a custom quantity field using list field type, and set 10 for each option (A, B and C) in the list. The field is used for both individual and group registration. I then went to register a group of 15. For all 15 members of the group, I was able to choose the same option (A), and was able to successfully submit the registration. I made another test, this time for 5 group members first, where I selected option B. I followed it with another group registration for 8 members, and selected the same option B. I was able to select the same option B for all 8, even when the 5 slots were already taken by the first 5 registrants.
The expected behavior is that it should not be possible to exceed the limit set in the quantity field, even for group registrations.