A very nice framework for web crawler and scrapping

Posted: November 28, 2010 in Web crawling
Tags: , , , ,

Recently I ran across a framework called Scrapy (http://scrapy.org/), which is a nice framework to acquire structured data from websites.  I recently used this framework to poll the CME FTP (ftp://ftp.cme.com/settle/) for settlements on the CME listed Eurodollar futures and it did very well and took considerably less software engineering than I would have thought.  If anyone is interested, I would be happy to share the Perl and Scrapy examples.

  1. KapnKrunch says:

    Do you mean share the python script? Perl or python, either way I would be interested to see your use of scrapy on CME FTP data.

  2. Mohan says:


    i am also looking for a high performance crawler framework in Perl. I come across Gungho from CPAN. Is Scrapy is also a good fit for high performance. Please share the code if possible.

  3. anteagle says:


    I am currently starting to use scrapy to crawl data from yahoo finance. I am interested in seeing your use of scrappy. Thanks if you can share the code.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s