Search API Notes

Scenario: You've got a great idea that requires indexing and/or search capabilities well beyond your budget.  Where do you go from here?

Thankfully, you have a few options to choose from when deciding how to power your new app.  Sadly, you have ONLY a few options to choose from.  Indexing and searching the Internet is a monstrous task, which is why this industry is a natural fit for the oligopoly we see today.  There are three players in this market that all offer Search APIs, but as of this writing, their products differ considerably.

Yahoo BOSS - http://developer.yahoo.com/boss/search/
If you are looking for something inexpensive, then this is it.  They offer a 'limitedweb' search that is slightly smaller and not as fresh, but it's only $0.40/1000 queries, which is half the price of their 'web' offering.  Other than the cost savings, this service stinks.  Do not use this unless your application allows for a large margin of error and cost is the most important requirement.  I've found 3 types of common problems:
- False positives: returning results that do not contain the query.  It doesn't matter whether you are using an exact phrase search, boolean operators, etc.  Regardless, you will get false positives from time to time.
- False negatives: matching results that are in Yahoo's index fail to be returned sometimes
- Sporadic errors: the errors mentioned above, as well as other outages, occur frequently and randomly.  While developing with this API it was very frustrating because it does not return consistent results.  The same query will return no results one minute, then many results a minute later.  Frustrating.
Bottom line: DO NOT USE ON IMPORTANT WORK

Google - https://developers.google.com/custom-search/

On the other end of the spectrum is the dominant search giant.  Their API is high-quality and VERY expensive ($5/1000 queries).  Notice that that is more than 10X the cost of Yahoo's limitedweb queries.  Nevertheless, the Google results are consistent and of the quality you would expect.
Disadvantages: Besides price, Google's API results often do not match their public search results.  If you have a high volume app, the rate limits may be a deal-breaker for you (it was for us).

Microsoft Bing - http://datamarket.azure.com/dataset/8818F55E-2FE5-4CE3-A617-0B8BA8419F65
I'm rarely a fan of anything Microsoft produces, but they are the winner in my evaluation of web search APIs.  They have just the right mix of consistency, price, and performance, without the restrictions of Google.  They offer unlimited searches at a price that is roughly $1.25/1000 queries.  This is 1/4 of Google, but still 3X more than Yahoo's limitedweb.  For mission critical apps that can't afford the problems of BOSS, Bing is probably the best choice.  Be sure to use the "Web Only" API if you are only using their web search, as it is cheaper than their composite search offering.

Resources for the Internet of Things

Whether you refer to it as The Internet of Things, the Sensor Revolution, or the Programmable World, here are some potentially useful resources:

• OpenRemote
• SmartThings - Developers and Store
• MakerSwarm
• RaZberry - Z-wave for Raspberry Pi
• Nimbits
• ioBridge - complete stack 
• NinjaBlocks - sensors, devices, and programmable hub
• Limitless LED - programmable lights
• WEMO - Belkin home automation devices
• Alyt - Android-based home security and automation hub & devices

Microcontrollers & Systems on a Chip

• TinkerForge - Stackable, programmable boards.  Appears to be simpler than Arduino in many ways.
• Arduino - Everyone's favorite open-source microcontroller
• BeagleBone Black
• Raspberry Pi
• Galago -
• Intel Galileo - open-source board from Intel

I will try to expand this list over time.  If you have more, please add them in the comments.  Thanks!

Will a Robot Take My Job?

With many jobs lost and the economy teetering on recovery, this question couldn't be more relevant. The man vs machine debate has a long history and conjures up images of John Henry racing to his death against a steam hammer. The reality is that machines already do the work of people and will continue to usurp greater and greater responsibilities. If your job is not one of the ones lost, then it is quite easy to see that this expansion should be welcome as it allows for greater societal productivity. After all, do you miss doing dishes by hand? Wish you could wash your laundry in a tub and hang it out to dry? Want to pay greater prices for hand made products? The key is understand which jobs are likely to be filled by machines. The following 5 questions should help.

  

Is your work highly repetitive?  
Specialized, repetitive tasks are easier to automate.  It is the human ability to generalize that makes our human "wetware" different than robotic hardware. Maybe you execute the same movements as part of a manual labor position, or maybe you spend the day downloading the same data and plugging it into the same spreadsheet.  Whatever the case may be, if you find yourself doing the same tasks over and over, then your job is more likely to be taken by a machine.


Are there many people in your company in the same role as you?
If so, you are a great target for automation.  Developing algorithms and/or hardware to replace one person is often not cost effective, but it may be a no-brainer for a company to invest in machines that replace 100 people.


What collar do you wear?  White or blue?
Previous generations of machines have found great success replacing manual labor in factories, but the next generation of machines will be more likely to replace office workers -- white collar folks.  The reason for this is that massive data and computing power have reached a critical mass allowing difficult "thinking" tasks to be conquered by machines.  We now have machines that can dominate Jeopardy, diagnose cancer from a breast scan, and grade essays.  This trend will only increase as more data and computing power become available in the years ahead. 

On the other hand, most jobs that involve service work or manual labor will be difficult for robots to replace.  The reality is that complex analytical thinking will be easier to duplicate than our nerves and muscles.  We take for granted how easily we handle sophisticated movements, but it will be many years before robots are equipped with the sensors and actuators comparable to a human's.  Until then, don't expect the moving company to send in robots to pack up your house.


Do you interact with people?
Many jobs require a uniquely human touch, something that machines do not offer -- something that machines may never offer.  If meaningful interaction with people is an important part of your job, then you are unlikely to be replaced by a machine any time soon.  Don't expect to be seeing a salesperson or a robotic shrink in the next century.




The bottom line is that the use of machines will continue to allow society to be more productive, and productivity often means one person doing the work of many.  We should embrace this new potential, but also be cognizant of the skill sets most needed in the 21st century.