Archive for the ‘Web crawling’ Category


Hola!

I would like to share with you a small project that I created in C# for collecting end-of-day data from Yahoo!Finance.  This application (source included) will create a directory on your C:\ drive called “data” where it will create many CSV files for each instrument listed in “MasterSymbols.txt” listed in the Working directory, that is the directory that you run the executable from.  Add symbols to the “MasterSymbols.txt” file if you want to have the application pull them down as well. It is a console application that will write out exactly what symbol it is downloading and from when it is doing this.

The source code and binaries are here

You will need 7-zip to un-compress them.


Recently I ran across a framework called Scrapy (http://scrapy.org/), which is a nice framework to acquire structured data from websites.  I recently used this framework to poll the CME FTP (ftp://ftp.cme.com/settle/) for settlements on the CME listed Eurodollar futures and it did very well and took considerably less software engineering than I would have thought.  If anyone is interested, I would be happy to share the Perl and Scrapy examples.