Abu Haider Siddiq

Install a thirdparty library: sudo apt-get install acl
Then, in following command, change the username to the user you want to restrict the access:
sudo setfacl -m u:username:--- /path/to/directory