--- title: "[New article]: Creating a generic Site-To-RSS tool (site scraping)" url: "https://osherove.com/blog/2003/9/28/new-article-creating-a-generic-site-to-rss-tool-site-scrapin.md" canonical: "https://osherove.com/blog/2003/9/28/new-article-creating-a-generic-site-to-rss-tool-site-scrapin.html" published: "2003-09-28T20:48:00" updated: "2003-09-28T20:48:00" tags: [".Net Original", "Cool Articles", "Cool Tools"] author: "Roy Osherove" description: "It's been a long and very interesting week for me. I know it's weird hearing that the week I start looking for a new job, but it's true. I've been getting lots of interesting offers from people, especially from the…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # \[New article\]: Creating a generic Site-To-RSS tool (site scraping) By Roy Osherove September 28, 2003 In [.Net Original](https://osherove.com/blog/category/net-original), [Cool Articles](https://osherove.com/blog/category/cool-articles), [Cool Tools](https://osherove.com/blog/category/cool-tools) **Historical article** · Originally published September 28, 2003; technical details may be outdated. It's been a long and very interesting week for me. I know it's weird hearing that the week I start looking for a new job, but it's true. I've been getting lots of interesting offers from people, especially from the weblog scene. I'm still looking, though, so keep'em coming. If you want me - [tell me](http://weblogs.asp.net/rosherove/posts/6870.aspx)! (hint, Microsoft!) The past week has given me some time to relax and start doing the things I really love - late night coding and discoveries. That's exactly the case with the article I'm presenting here. I was sitting at my machine, browsing [.NetWire](http://www.dotnetwire.com/) when I suddenly thought, “gee I **wish** they had an RSS feed!”. Then I started playing around with some regular expressions using [eXpresso](http://www12.brinkster.com/ultrapico/Expresso.htm), and it dawned on me that it would be real easy to engineer a generic solution for such a problem. So I did. And here's the product of this work; The longest and most detailed article I've ever done (about 5,000 words - personal record). Yeah, length doesn't matter, content does, but there's a lot of content in there. This really is something I plan to pursue further. Thanks to [Mike Gunderloy](http://www.larkware.com/) for helping me out on a nasty bug I couldn't figure out. You saved me, buddy! Anyway, I'll stop rambling and give you the juice: [Creating a generic Site-To-RSS tool](http://weblogs.asp.net/rosherove/posts/29509.aspx) **Summary:** _I’ll show how to use regular expressions to parse a web page’s HTML text into manageable chunks of data. That data will be converted and written as an RSS feed for the whole world to consume. Finally, I’ll show how to create a generic tool that allows you to automatically generate an RSS feed from any given website, given a small group of parameters. At the end of the day we will have a working RSS feed for_ [_www.DotNetWire.com_](http://www.dotnetwire.com/) _._ [Read it](http://weblogs.asp.net/rosherove/posts/29509.aspx). Have fun. Be productive. **Disclaimer:** Site scraping personal use only! They already have an [RSS feed here](http://dotnetwire.com/newsfeed/rss/) (which was mentioned to me only after the article was finished) **P.S** Just realized this is the 200th article on .NetWeblogs! Is there a prize? :) **Update:** Well, ScottW posted that: 1\. They already have a feed (so joke is on me!) 2\. Scraping a site is stealing page views! so do it for personal pleasure only! ## More in .Net Original [All posts in .Net Original](https://osherove.com/blog/category/net-original) - Sep 30, 2003 ### [See you at the Israel .Net user group tomorrow!](https://osherove.com/blog/2003/9/30/see-you-at-the-israel-net-user-group-tomorrow.html) - Sep 30, 2003 ### [Unit Test Your .NET Data Access Layer: Where are the Mocks?](https://osherove.com/blog/2003/9/30/unit-test-your-net-data-access-layer-where-are-the-mocks.html) - Sep 25, 2003 ### [AmazReplacer - Convert Amazon's URL into your referrer URL automatically](https://osherove.com/blog/2003/9/25/amazreplacer-convert-amazons-url-into-your-referrer-url-auto.html) - Sep 24, 2003 ### [\[Article\] Fun with Unit Tests – Testing abstract classes](https://osherove.com/blog/2003/9/24/article-fun-with-unit-tests-testing-abstract-classes.html) --- ## About the author **Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: https://x.com/royosherove - LinkedIn: https://www.linkedin.com/in/osherove/ ### Contact - Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7 ### License Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/