Selenium is a browser automation framework. There is a getstarted tutorial at Selenium wiki that looks like a good place to get started. Since firefoxdriver is a more complete solution comparing with HtmlUnitDriver due to the fact that javascript will get executed in a browser, I will just skip the HtmlUnitDriver part.

Of course, we need to find the maven dependency for Selenium where you can find it here. I am planning to use 2.39.0 in this case because it seems like it has the highest adoption rate.

Here I created a Java class, which has a method that will take in an URL and return the HTML source code of that page. Of course, since Javascript execution will take time and you have to give the browser a signal of when should be the success of the fetching, in my case, is when the browser is able to find an element that matches a customized xpath. If not, it will try to wait for a certain amount of time.

And here is how you can grab a webpage in one line using Selenium.

Of course, there are tons of things that need to add into this protocol like error handling etc.

But at least, we have a straw man right now!