word.camera generates paragraphs from a photograph

April 13, 2015 7:20 AM

word.camera generates paragraphs from a photograph
With Clarifai, an image concept extraction API utilizing convolutional neural networks, and ConceptNet, a lexical relationship database, I built a template system to generate paragraphs of text from photographs. word.camera is responsive — it works on desktop, tablet, and mobile devices. The code behind it is open source and available on GitHub, because lexography is for everyone.

lexograph /ˈleksəʊɡɹɑːf/ (n.)
A text document generated from digital image data
___

Users can share their lexographs using unique URLs. Of all this lexographs I’ve seen users share since the site launched, this one stuck with me the most.

At the moment, the results are not nearly as grammatical as I would like them to be, and I’m working on that. The algorithm extracts tags from images using Clarifai’s convolutional neural networks, then expands those tags into paragraphs using ConceptNet (a lexical relations database developed at MIT) and a flexible template system. The template system enables the code to build sentences that connect concepts together.

This project is about augmenting our creativity and presenting images in a different format, but it’s also about creative applications of artificial intelligence technology. I think that when we think about the type of artificial intelligence we’ll have in the future, based on what we’ve read in science fiction novels, we think of a robot that can describe and interact with its environment with natural language. I think that creating the type of AI we imagine in our wildest sci-fi fantasies is not only an engineering problem, but also a design problem that requires a creative approach.

I hope lexography eventually becomes accepted as a new form of photography. As a writer and a photographer, I love the idea that I could look at a scene and photograph it because it might generate an interesting poem or short story, rather than just an interesting image. And I’m not trying to suggest that word.camera is the final or the only possible implementation of that new art form. I made the code behind word.camera open source because I want others to help improve it and make their own versions — provided they also make their code available under the same terms, which is required under the GNU GPLv3 open source license I’m using. As the technology gets better, the results will get better, and lexography will make more sense to people as a worthy artistic pursuit.

Role: programmer

posted by TheMadStork (4 comments total)

This project was posted to MetaFilter by joseph conrad is fully awesome on April 27, 2015: Then, a sky, an urban, and an empty. Here, the sky is for fly in.

For me, this is a great tool to generate lots of possible creative seeds from a source image. So Thanks!

But also, this is the best commentary I've read so far on Hillary's announcement to run.
posted by Kabanos at 8:13 AM on April 13, 2015 [3 favorites]

I actually saw this linked in my email this morning, courtesy of Alexis Madrigal's awesome Real Future newsletter. Tried it out and found it a really interesting experience -- in fact, I immediately thought of posting it to Metafilter! At first blush it's almost impenetrable free-association like a stuttering thesaurus, but the more I used it the more I enjoyed it. Excellent work!
posted by churl at 10:22 AM on April 13, 2015 [1 favorite]

This is fascinating and eerie. Maybe it could write a picture book for this year's nanogenmo.
posted by moonmilk at 6:10 PM on April 15, 2015

It'd be nice if the word.camera site could include a couple of links to sample output. I found the site a little "so what is it then?" opaque; but then found the two examples linked above fascinating. This is very cool.
posted by We had a deal, Kyle at 2:18 PM on April 22, 2015 [1 favorite]