Why Requests Is Not Working Well As Well As Urllib2

GF: hey baby, can you help me get the pricing for my wishlist?

BF: absolutely, I can help you scrape lancome’s whole websites and give you all the pricing! (fingercrossed).

I took a quick look at lancome’s website and realized that it is a very typical website, every tag has well formatted class name or id. Even if they don’t have well formatted json requests while monitoring their network activity, it is still doable just collect the populated HTML and use beautifulsoup to narrow down to the desired elements and content.

lancomewebsite

I used urllib2 to make a simple http request and clearly what I want is captured in the response and then it is just a matter of parsing the HTML, thanks to Beautifulsoup, life is so easy now.

Then the idea of using requests quickly came up to me. However, seems like the requests library is not working right out of box.

Here is a screenshot of comparing urlib2 with requests.

lancome_urllib2_requests

In the end, I realized that `r.text` is not enough. First, r.text will return the html in the unicode format and then screw up the soup while passing to the BeautifulSoup function.

In the end, I have to upgrade my Python version and meanwhile using r.content instead of r.text to make it work.

VISA – Permanent Labor Certification Detail

In the big data world, qualified data scientists are really rare and within the qualified ones, most of them are well educated, highly skilled foreign labors, like people from China, India etc.

For the alien employees to legally work in the U.S., they have to hold certain type of VISA (H1B temporary working VISA). To make sure they will keep contributing to the employer, one must understand the PERM process and help the employee to get GC (green card) so the employee can work permanently in the U.S.

Here is a page from United States DOL (Department of Labor) that briefly describes the 9 steps to get the PERM application approved.

http://www.foreignlaborcert.doleta.gov/perm_detail.cfm

perm

SCRUM – Free Online Strum Training Series

Here is a list of free online scrum training videos. They include a brief introduction to the idea of SCRUM as a whole, and then dive into each component of a scrum process. They have some interesting animations and real life case which makes the training very logical and vivid. I personally think this is the most amazing and well-made online scrum training. Definitely check it out if you think your team is not well organized.

scrumtrainingseries

scrumtrainseries

Python – something about function arguments

We have talked about briefly how to read in arguments from the command line using argparse from this post, however, do you know there is really a lot that you might not know about defining a function in Python. How to read optional argument, how to read in positional argument and also how the scope/namespace works. Those are all really interesting questions and you can check out this python documentation – Controlflow.

pythonargumentsThe example above show that an argument start with one start will read in the arguments sequentially and it doesn’t have to use the keyword args even if most people use *args all the time.

Second, **kwargs is short keyword arguments. As you can see, it will parse the function input as a dictionary and store the keyword along with its corresponding values in a dictionary.