Blogdigger Dev Blog

Blogdigger Acquired by Odeo

2008-03-18T15:48:00.005-05:00

I'm very happy to announce that Blogdigger has been acquired by Odeo. Over the last few months, I've been working with the folks at Odeo in incorporating Blogdigger's aggregation and search technology into Odeo's new beta site. Along with the acquisition, I've joined the new Odeo team as Vice President of Search and Engineering. Basically, I'm doing what I did here, only I don't have to pay bills. ;)

A little about the new Odeo: it's focused on much more than just audio podcasts; we've added video and are pulling content from a variety of sources, not just podcasts. I'm really excited about what we're building and the team we've got(which includes the folks from FireAnt and the Odeo guys) is really smart and great to work with. If you'd like to check out the new Odeo, head over to the site and signup for the beta (and ping me and I'll approve your invite).

An additional reason I'm really excited about Odeo: while Odeo started off as a west coast company, it now has a really strong east coast presence, the DC area in particular. I'm hoping that we can play a part in what I think is going to be a big year for the DC area in terms of growth in social media and consumer web ventures.

As for Blogdigger: don't worry, it's not going anywhere. Blogdigger will continue to operate as it always has, providing current, spam-free blog search results. You can still contact me with any questions, and we hope to be making some improvements to the Blogdigger services after we've got the new Odeo up and running.

I know it's getting a little crowded down there, but look for a new logo along side our spotted donkey on the home page in the next few hours!

Database Maintenance

2008-01-25T10:20:00.000-05:00

As we mentioned before, Blogdigger's database has suffered a major hardware failure, and we need to take the database offline for some repairs. We're hoping this will cause only a minor interruption of service, but this is a somewhat complex operation. We estimate that the site should be back online by early next week, if all goes well. Thanks for your continued patience, and we hope to be back up and running soon!

UPDATE: The database rebuild process is going well; we've got the new database software installed and have begun reimporting the data. Some more recent data is already back up, so search results should show on the site. We hope to complete the restoration in the next 24-36 hours, then we'll restart our crawler to begin adding new content. Hopefully by Monday things will all be back to normal.

One note: there are some duplicate search results showing for some more recent blog posts; we are aware of this and will have this fixed as well by the end of the weekend.

Blogdigger Status Update

2008-01-18T13:16:00.001-05:00

Hey y'all. I know it's been quiet here for a while, but that doesn't mean things aren't going on. I wanted to update you on Blogdigger's status, as we've had a bit of downtime lately. We experienced a drive failure on our main database server that has caused a bit of trouble. We're in the process of rebuilding it, and we expect that things should be back up online in a few hours.

Many folks have emailed me about groups not updating for a week or so; this is a separate issue, but one that we're working on as well. We're hoping that groups should begin updating again over the weekend, if all goes well. We apologize for th inconvenience and appreciate your patience.

Stay tuned for more exciting news over the next week; we're hoping to put these hardware woes behind us and share some good news with you soon.

Get Blog Search on Your Site - Blogdigger Partner Feed

2007-06-12T12:55:00.000-05:00

We're happy to announce the official launch of the Blogdigger Partner Feed program. By joining the Partner Feed program, you get access to an XML feed of blog search results from Blogdigger, to incorporate into your application, site or search engine. Bring the full power of Blogdigger's blog search engine integrated into your site.

Check out our page on the Blogdigger Parter Feed Program, and join the growing ranks of sites that are adding blog search and content. Sign up today!

Updates to Blogdigger Search Feed Subscriptions Format

2007-06-05T10:24:00.001-05:00

One of Blogdigger's most useful features is the ability to subscribe to a feed for any search. Just do a search on Blogdigger for anything you are interested in, and look for the big organge feed icon over on the right; you can subscribe to a Blogdigger search in your favorite RSS or Atom aggregator, and when Blogdigger finds new blog or news items matching your search criteria, you'll automatically get it in your reader.

We've updated the format for the feed search results; as always, we provide an excerpt of the post, and a link back to the original site. We've added in additional information, such as tags, as well as a link back to the Blogdigger search results page (in case you check up us and see how we're doing). We hope this will make our feed subscriptions more useful to you in providing you with quick and easy access to more information.

If you'd like to use Blogdigger's search results in an application, or display them on your site, you'll need to get set up with a Blogdigger partner feed. All you have to do is contact us, and we'll make it simple for your to get our high-quality, spam-free blog search results incorporated into your site or app.

AltSearchEngines.com - A Blog for Alternative Search Engines

2007-06-04T08:50:00.000-05:00

Charles Knight of Read/Write Web has been a constant advocate of the work done by alternative search engines, publishing his monthly list of the Top 100 alternative search engines and following news very closely. Now Charles and the Read/Write Web team have launched a new site, AltSearchEngines.com, a blog devoted to covering the ins and outs of the alternative search engine space. Check it out!

TECH Cocktail DC

2007-05-16T18:51:00.000-05:00

Blogdigger is proud to be sponsoring tomorrow (May 17th) night's TECH Cocktail event in Washington, DC. The TECH Cocktail team has done an amazing job in getting together a really great group of people, I'm really looking forward to meeting everyone from the DC tech community.

I'll be there presenting Blogdigger, along with Michael Miller, who will be introducing something new (and very exciting) so make sure to come over and check it out. You can still register if you haven't yet, it's free, and there's a whole bunch of great people attending. See you tomorrow!

PC World's Search Engine Shoot-Out

2007-04-25T14:40:00.000-05:00

PC World just published the results of their Search Engine Shoot-Out, comparing search results from various search verticals including video, images, blogs and local information. I'm very happy to report that Blogdigger faired fairly well, finishing third in the blog search category. While I'd obviously like to do better, we managed to beat out some of the more well-known players, which I'm happy about.

There is some good feedback in the article: PC World really liked our clean search results page; we've intentionally aimed at having a clear results page, and we're working on adding in access to our other search applications. I want to be able to get our blog, media and local results all on a single, clean results page, and that's what we're working on. On the negative side, we got dinged for being a bit slow to index new blog posts, which is a fair criticism, and why we're hoping to launch our new and improved crawler in a few weeks, which should pick up some of the slack. In the mean time, the best way to get your posts indexed as quickly as possible is to ping Blogdigger when you publish.

Make sure to check out the whole article, PC World's done a really good job in covering all the major search verticals. Take a look at the blog search results, as well as the test criteria (very well done, in my opinion). And thanks to PC World for including Blogdigger in the Shoot-Out!

Blogdigger Makes List of Top 100 Alternative Search Engines

2007-03-29T11:34:00.000-05:00

Thanks to Read/Write Web's Charles Knight for adding Blogdigger to their list of Top 100 Alternative Search Engines for March 2007. They listed us as new, which we'll assume means new to them. In all seriousness, we're extremely honored by this distinction, so thanks Read/Write Web!

Make sure to check out the whole list, there's lots of interesting stuff going on in search. If any of the other top 100 want to add blog search to their sites, just contact me and we'll give you free access to Blogdigger's partner program for a month, which means you can syndicate our search results onto your site and offer blog search to your users.

Controlling your Online Identity

2007-03-27T12:14:00.000-05:00

ComputerWorld has an article entitled Web anonymity can sink your job search, which explains the benefits to having a well-controlled online identity; one of the best, if not the best, ways of doing this is by publishing your own blog, so that when recruiters, or anyone, for that matter, checks you out in Google (and everyone is doing it), they see what you want them to see, not what others may have said. Of course, sometimes you want to blog and NOT associate your identity with what you are writing about; the anonymity of the Internet allows one to participate in conversations that are of interest, but might not be related to one's professional identity. In such a case, it's important to take careful control of how your blog and personal information are presented online.

One of, if not the, most frequent support requests we get is from bloggers who's real-world identity has been associated with their personal blog. Doctors, lawyers, teachers, mothers, priests, rabbis; you name it, they've contacted us and asked to have their real names removed from Blogdigger. We are certainly happy to comply, we want to do the best we can to protect a blogger's privacy. But in almost every case where we are contacted, the same thing has happened: the blog hosting software includes the bloggers real name by default in the blog's syndicated feed. This tends to be an issue with paid blogging services where a user is required to enter contact information for payment purposes. When services that feed off blog's syndication feeds, such as search engine like Blogdigger, or aggregators like Bloglines, the author's full name is available and searchable.

Often, these blogging services have settings to publish a nickname instead of the user's real name, but the nickname setting is not the default. When setting up a blog, or taking any action online, it's important to take into account how you are associating your actions with you personal identity, and to do that, it's important to understand how blog service providers work, and what tools are available to help track your online identity. A mistake could be around for a while.

Dusting Complete...

2006-12-15T09:06:00.000-05:00

Just to let you know, we've finished our maintainence, so things should back to normal, thanks for your patience. Dig in!

By the way, did you know that the best way to ensure that Blogdigger indexes your blog as soon as possible is to ping us when you update? You can do this by adding http://www.blogdigger.com/RPC2 to your list of ping servers (if you use Wordpress or MovableType). If you use Blogger or another service that does not allow you to configure your list of ping servers, you can use a service like Pingoat to create a bookmark that you can click on each time you update.

Pardon the Dust

2006-12-14T09:22:00.000-05:00

We're in the process of an index upgrade, so Blogdigger might be feeling a bit sluggish today, and new updates may take a while longer to surface in the index. The good news is that when it's through, it will mean some pretty nice performance enhancements on our end. So please bear with us, things should be back to normal later today. Thanks!

New and Interesting Groups

2006-12-06T13:21:00.000-05:00

One of the greatest things about blogs is the diversity of topics on which they cover. It never ceases to amaze me the broad and far-reaching topics discussed with great detail and passion on blogs, and their ability to foster communication and interaction within a community.

Today I noticed a relatively new Blogdigger Group that I thought exemplified this; it's called the SLDevelopers Forum, it's an aggregation of feeds and blogs discussing developing for Second Life. Very cool stuff. There's a ton of other interesting groups out there as well.

Diggers on the go!

2006-11-10T13:08:00.000-05:00

If you were at last Sunday's New York Marathon, after looking for Lance Armstrong, you may have noticed a lone runner sporting a Blogdigger t-shirt. We like to spend our marketing dollars creatively, and so we sponsored our good friend Ari Schulman (and "sponsored" here means, we got him a Blogdigger t-shirt).

What makes this significant is that Ari had been training for several months to run the marathon, and two weeks prior, he was admitted to the ICU with Guillain-Barré syndrome. We were all concerned he wouldn't be able to run, but Ari pulled through and made an amazing recovery, and in just two-weeks time was back on his feet, and completed the marathon in good time (he didn't keep up with Lance, but he did just fine).

Our very own Mike Miller also ran the Marine Corp Marathon a few weeks ago. Mike didn't have any celebrity runners to keep pace with, but did finish strong.

Congratulations Ari and Mike, and thanks! Diggers always go the long haul!

DNS Issues

2006-11-02T01:52:00.000-05:00

We're having some DNS issues, things should be straightened out shortly. Sorry for the inconvenience.

Stay Tuned

2006-08-18T12:42:00.000-05:00

Things have been quiet on the Dev Blog for a while; rest assured we've been hard at work on a number of fronts. The main focus of the past few months have been infrastructure improvements (which have increased our stability and uptime tremendously) and focusing on revenue streams (I'm very happy to say we are profitable, have been for a almost six months now, and getting better every day). Over the next few months we will be refining our user interface to clean up things and make the site overall more usable (and I'm looking for a good CSS/Javascript/front end developer familiar with JSP; so please get in touch if this is you).

Over the coming week I'll be posting some information on our existing partner program and new advertising opportunities, so stay tuned.

I'm going to be in the Bay area next week, so drop me a line if you want to connect.

DC 2.0

2006-03-13T11:43:00.000-05:00

If you’re in the DC area, make sure to sign up for DC 2.0, scheduled for Wed. March 15 in the Reston offices of Mintz Levin. Blogdigger is proud to be sponsoring the event, and I’ll be there to demo our service. More importantly, I’m hoping that this will be one of many future DC-area tech/geek/Web 2.0 gatherings.

Tags: Web2.0, DC, Blogdigger, DC2.0, Events

Who wants lunch?

2006-02-27T22:21:00.000-05:00

Check out lunchSpark, a new social network app with a focus on one of my three favorite meals of the day (lunch). lunchSpark has a comprehensive list of eating establishments searchable by location; after you create a profile, you can see who else is in the area with similar interests, and hook up for a meal.

lunchSpark wanted to include supplemental information on each restaurant, whether it was a national chain or a local establishment. In order to do accomplish this, lunchSpark is incorporating user-generated content accessed through Blogdigger's search API. By pulling a search result feed formatted in RSS, lunchSpark was able to quickly incorporate highly relevant blog content into their site.

Here's my profile on lunchSpark (if you're in the area, let me know, I'm often free for lunch), and here's the page for Max's, home of the best schwarma in all DC (scroll down for the Blogdigger entries). I'll eat there anytime.

Welcome Podloco!

2006-02-21T14:40:00.000-05:00

Blogdigger is proud to be working with Podloco, a resource site for podcasts with a local focus. Podloco helps publishers create and publish their podcasts and connect with their target audience. Podloco has an extensive directory of podcast content, which is powered by Blogdigger's Media Search.

Podloco needed a quick and simple way to get relevant content for their podcast directory, without going out and finding podcasts, indexing their content and building a search engine for them. Using Blogdigger's partner feeds, Podloco was able to quickly populate their directory using a flexible keyword taxonomy. Since Podloco's directory is based on keywords, they can update or add to their directory at any time and Blogdigger's Media Search categorization takes care of the rest.

Check out Podloco, and if you're interested in working with Blogdigger, please, let us know!

Four Things

2006-02-21T14:01:00.000-05:00

I'm sure you've chanced across the Four Things meme making its way around the blogosphere. I was quite touched to find a few folks, Jeremy Pepper and Andrea Weckerle to be specific, who included Blogdigger on their list of sites they visit every day! Thanks guys, if you ever need anything, just let me know!

Do you use Blogdigger? Don't keep it a secret, tell a friend and spread the word.

New Advertising - What do you think?

2006-02-16T22:10:00.001-05:00

We're testing out some new advertising on our search results pages. In place of sponsored links along the right-hand side of the results, we're displaying image ads. We're using a new advertising system, so please let us know what you think; we want to make sure the advertising is complimentary to the rest of the site experience.

We'll be testing several different advertising options over the next few days/weeks, so please let us know your thoughts. Thanks!

Looking for Feed Feedback

2006-02-10T14:14:00.000-05:00

We recently pushed some updates to our feed advertising system. The system is designed to ensure that an ad is only displayed at specific intervals. It seems that the changes may have thrown a monkey wrench into things, and some folks are reporting multiple views of the same ad. I'm hoping that this will stabilize over the next few days, but if you are seeing weird behavior, please get in touch so we can do our best to fix it. Users of Thunderbird are of particular interest, as we've gotten a number of reports from folks using Thunderbird of seeing multiple ads with no new posts.

Thanks!

FeedFlare API

2006-02-09T10:58:00.000-05:00

FeedBurner just announced their FeedFlare API, which lets third parties (like Blogdigger) integrate their services so FeedBurner users can add them to their feeds.

I did some quick coding and added link counts via Blogdigger's Link search to FeedFlare. You can see an example of what the result looks like by taking a look at my personal blog's feed.

To add Blogdigger Link Search counts to your FeedBurner feed, add the URL

http://www.blogdigger.com/bd-linkflare.xml

as a custom FeedFlare unit on your FeedBurner FeedFlare page (you have to log in to FeedBurner, go to Optimize, then FeedFlare).

I've got a few more FeedFlare's planned...the next one will likely involve FeedBurner's geotags and Blogdigger Local...stay tuned. If you've got other ideas for how to integrate Blogdigger's data into FeedFlare, just let me know!

On Performance and Scalability

2006-02-01T14:28:00.000-05:00

If you're an avid follower of the all-seeing, all-knowing GrabPerf, or just use the Blogdigger web site a lot, you may have noticed we've been sucking wind quite a bit lately. A number of factors have combined to push usage of Blogdigger to new levels. Although I haven't blogged much about it, addressing this rapid growth has been pretty much the main focus of my time over the past two months. We have made several improvements, each of which, for one reason or another, soon became irrelavent as site usage continued to increase*. We pushed out some changes last night, which seem to have made things happy for now (fingers crossed, no jinxes); Feb. 1 was our highest traffic day yet, which says to me that if we can keep the site up and running, we've got an audience just waiting to use the service.

Folks talk about scalability, etc., but the truth of the matter is, when you're running on limited resources (Blogdigger currently runs on two boxes), you can design your software to be as scalable as you like, but until you have the money to grow your infrastrucuture, it's a bit of a moot point. I'd argue that is a good thing; it's easy to take a system, throw more hardware at it, and walk away without really attempting to understand where things are going wrong. The first step in addressing scalability should be to look at the application, your deployment, configuration, etc., and see if you can improve things without adding more hardware. From what I've seen, most problems in software development come from misuse or misconfiguration of a software component, rather than limited resources. In other words, sometimes you don't need another database server, you just need more database connections in your pool.

Just so you know, we're continually monitoring the site for performance, and are always looking at ways to make things better. If you have ideas, questions, comments, please don't hesitate to let us know.

* - As an example, after we made a performance enhancement sometime in December, Bloglines started up their new crawler. Bloglines new crawler is a hungry little bugger; it likes to visit each of it's feeds twice an hour, independant of any other variables such as whether the feed has recently updated/pinged, how frequently new stuff is available on a feed, or what the ttl value for the feed is set to. Another example, just a few days after we pushed out a really nice performance enhancement (Steven from Grabperf even emailed me to check if we'd gotten more boxes) one of our partner sites, Webjay, went and got acquired by Yahoo.

Have you heard the word?

2006-01-29T22:54:00.000-05:00

It is with great pleasure I present gregword.com, my new personal blog. I've got a post up with the details as to what's up, so please head on over, give it all a look-see, subscribe, link, etc., etc. Satisfaction guaranteed, or your money back.

The Blogdigger Dev Blog will continue to update, but focused more on Blogdigger and blog search.