Almost three years ago I programmed a simple twitterbot (see here), namely a Python script that was posting every hour, when available, news or blog posts related to cultural evolution – hence the name @CultEvoBot. While the goal of the endeavour was mainly to see how difficult was to build something like that (it was easy!), and to use potentially what I learnt for other projects (I never did, but who knows!), @CultEvoBot was relatively useful and posted links to interesting sources, the majority of the time.
Initially, it used google news search or google blogs search (after having flipped a coin to decide), searched there for “cultural evolution”, and posted to twitter the first link which was not in its log file of past – already tweeted – links. Pretty straightforward. The first under-the-hood modification was to add other searches, as quite often nothing new was present for “cultural evolution”. I therefore added a part of the script that, when nothing was found, was searching for “cultural transmission”, “evolutionary anthropology”, or “cognitive anthropology”. If again, nothing was found, @CultEvoBot remained silent.
The second – and last – modification (this one visible, and completely useless), was to change the header photo (from the classic tree of life in the Darwin’s notebook to the Telephone Tower of Stockholm at the end of XIX Century – photo from the flickr account of Stockholm’s Tekniska museet, which I quite like, and I used for another post) and the profile photo from Steel Jeeg to Mazinger (yes, I was a child in the 80s, and not, I do not care at all about all this).
Over the time, @CultEvoBot twitted 6,319 times (plus one when I will have it tweet this post, by hand), and it gained almost 500 followers (489 at present). Truth be told, I’d say half of the followers have absolutely nothing to do with cultural evolution, but they are a result of another part of the script that was automatically following back the followers (the twitter timeline of @CultEvoBot was at time NSFW, for example…). Anyway I judge this a fair result, and, as I mentioned before, I believe it was providing a reasonably good service.
The problem arose when the Google Web search API (in fact, deprecated from 2010, my bad), on which @CultEvoBot was based, stopped to work at all, or did it very salutarily, few months ago. The replacement suggested by Google (Google Custom Search) does not seem to work as well as the old API – or it is entirely possible that I did not understand something, which is mainly why I am writing this post. The script should repeat exactly the same queries, and I used the schema.org system (as suggest by Google) to restrict the search to “NewsArticle” and “Blog” types.
Just to give you a feel of the problem, here are the last five links proposed by @CultEvoBot:
- tourindiaapp.com/tourindiaapp-blog/blogger/…/franklyndawbin0 (it looks like a page with random text, from a website proposing a “Tour India” application)
- deniasays.blogspot.com/ (is that Indonesian?)
- suvasishpoddar.blogspot.com/ (an almost acceptable personal blog of a guy. Not enormous academic value)
- www.everipedia.com/Traditional_transmission/ (what is everipedia?)
- ogena.net/Arts/Literature/Poetry/Map (this I can not even figure out)
And here, going to the beginning of the log file, the first five:
- http://www.wired.com/wiredscience/2013/10/peanut-butter-cookies-and-cultural-diffusio/ (a relevant Wired article from Samuel Arbesman)
- http://www.medicaldaily.com/rise-hpv-cancers-blamed-american-cultural-evolution-disparity-between-blacks-and-whites-puzzles (is the rise in HPV cancer linked to the 60s/70s sexual revolution? And why there are disparities between blacks and whites?)
- http://moreintelligentlife.com/content/arts/anonymous/xu-bing?page=full (a Chinese artist and his struggle with Maoism, I guess. Not perfect, but still…)
- http://www.forbes.com/sites/darwinatwork/2013/10/11/the-grand-theory-of-business-from-charles-darwin/ (Economy and evolution. Spot on)
- http://phys.org/news/2013-10-scientist-evolution-comfy-stage-threats.html (Press release from Harward, for a Daniel Lieberman paper)
The difference is, unfortunately, quite evident. In conclusion, for the time being, so long @CultEvoBot. If any reader has any knowledge of the Google Custom Search, or any suggestion for alternatives is more than welcome to comment here or contact me!
Also, if anyone is into text-mining and cultural evolution, here the log file of @CultEvoBot. It is a text file that collects 6,341 links (one by line) spanning a three-year period, that should link to 6,341 webpages relevant for cultural evolution (given what I said in the post, probably only around the first 6,000 are relevant). Again, happy to chat about this with anyone interested.