Recently I helped one of my client to setup a VPS on DigitalOcean (DO). The price is very competitive (the cheapest plan is $5 per month) and they offer SSD as the storage. I highly recommend to give a try if someone is looking for a virtual hosting.
Actually a VPS is just a server running on other location. Once you subscribe a plan and start to OS, it's your responsibility to maintain the system.
The steps below are to enhance the security once the server is deployed ( we are using Ubuntu 14.04 in this case). The objective is to harden the server so that it can only be accessed from a client with ssh
1. Setup ssh keyfrom a linux server at home or office. Assumed you have a linux client with a user "user1"