Viber offers a community that is controlled by the super admin or an admin. If you want to remove the member from chat, you can do it by following a simple procedure.
In this way, your non favorite contacts cannot join the group chat. To ban the members form the community you can open the community info from the main menu. Here you will get the option of showing all. Select it and click on Ban user.
Similarly, you can bring back the user to your chat list by editing to remove the user from the banned list. To unban members from the desktop version you have to select the "X" to remove the member from this list.
This is the simple way to remove contacts that you think annoy you or bring back the one who is important for your next meeting.