Lost In Translation

It is difficult to be perfect in any form of translation, and there is always information loss in translation. The loss leads to misunderstanding. Whenever we talk about translation, the most common example is language translation. For human, one of the difficulties of doing translation is synonym. There are many synonyms for one meaning, but actually these synonyms have nuance. As to computer, it is easy for the computer misinterpret the meaning.

What about series of translation? The first thing that come up to my mind is Telephone Game, a game that needs several people line up in a row and pass messages from the first one to the last person, and the message at the end is always totally different from the beginning.

It is difficult for us to do translation, how about computer? Machine learning is the buzzword for recent years, and it has done many good works that we used to think that can be only done by humans. However, we don’t really see a machine learning model has 100% accuracy or it cannot learn from new data and will be overfitting from the training datasets. Accordingly, there must be something lost during the process. Also, machine learning can be treated as a process of translation, so I want to see how would the computer do within series of machine learning translation.

Doing translation can be treated as the process of interpreting the target, and this is based on your knowledge and experience. If you ask different people to describe an image, a thing, or just repeat a story told by others, the answers are always slightly different since everyone has their own way to interpret. However, for something that we all know, although it’s not 100% the same, we can have consensus on what it is or the summary is. For computers, we give them training dataset to be their knowledge and experience. However, it is difficult to perfectly simulate human thinking and experiences. That is, I make this project to see how different would a computer and human interpret.

The process will start from a random image, and the user needs to describe the image in a sentence. The computer will draw a sketch based on the description. The computer will come up with a sentence from the sketch then generate a new image from the sentence. The user will describe the new image and make the computer do the same process again. The whole process will go for 3 rounds. At the end the user will see the whole translation story and they can see how the computer interprets the user’s description into a sketch and even a new image.

*User testing*


There are some approaches that I have done but did not use at the end. I have tried Detectron to detect objects from an image, but at the end I make the user to get involved with the process so I don’t need to make the computer interpret the image. The user will describe the image instead.

Another technique is sketchyGAN, it can generate an image from a sketch, however right now it can only generate the image from a sketch with single object, and the sketch generate from my process has multiple objects in it.

Github link

Data Art Project 1: Self Portrait

After coming to New York, I spent much more time on walking than I did when I was in Taiwan where we always ride scooters for moving.

So I get the data from my iphone by following this page introduced in the slide. Surprisingly, I found that my phone recorded my walking data from last year, which is about the time I came to New York.

However, the data only records my steps and distance, so drawing out my path is kind of impossible; moreover, I come up with a thought that how far I can go if I add up the distance I have walked.

So I use Mapbox as my “container”, and try to draw circles to represent the distance I walked in each month. For the data, I have to convert the distance (in miles) into pixels that Mapbox needs by following this stack overflow page. Also, I animate the circle I draw by following this example.


The full code is here.

Data Art HW1

For the first homework, I tried to find some libraries to visualize the immigration data.

I am still using the data in github so I think it is the immigration data of entire US.

For my first 2 charts, I used a library called Chart.js, which I think is a pretty handy api that can do some simple charts with few animations and decorations.

So I tried to make a bar graph and a pie graph. Since there are over a hundred countries, I feel that it will be better to put them into separate graphs in different regions. Fortunately, both of these 2 graphs are using similar arrangement of data, so that I only have to arrange the data once.

Due to the way of specifying data in Chart.js, I have to put all the labels(countries) into one array, and data(estimates and margin errors) in separate arrays.


For the third approach, I found an api made by Google called Treemaps.

The data should be placed like this:

It is pretty different from what Chart.js wants, so I have to do different arrangement to the raw data.
Also, it needs to specify the parent node of every country, so that it will have layers in the map.
The size of the rectangle will be determined by the proportion of total # of immigrations, and the color of each small part will be determined by the error data.


For me, I think the most important thing of doing visualization is to arrange your data that can fit in whatever libraries or whatever approaches you want to do. With proper preparation of your data, other decorations are the minor problems.

The full code is here

NOC Final Project: Style Transfer Chrome Extension

Rather than making a webpage and upload images by the user, I found that directly apply style transfer to Instagram images will be more fun. So I tried to combine the knowledge I learned from Hacking The Browser to make a chrome extension.

For this project, I think it can be separated into three parts:

The first part is train my own style base model. Originally I wanted to follow this repo to train my own model. However, it takes too long for me and I cannot make my own style model, so I can only take three trained model from this repo.

The second part is to do style transfer. Thank to wonderful ml5, it gets the jobs done easily. However, I don’t know if the style transfer function in ml5 has a little problem on getting image width and length. Because I found that the if the image is not a rectangle image, the output image will become a mess and also the width and the length of the output image seems to be swapped.

The third part is to make a chrome extension. Since I am going to get the image data from Instagram, there is a cross-origin data issue due to network security, so I have to find other approaches to get the image data.

I tried to download the image I chose so that the image I am going to deal with will become a local data. At first I thought the extension will pack all images onto chrome so the web accessible resources (images) will also unchangeable. Luckily it seems that it will still change if I download the image to my extension folder.


So downloading the image is somehow a little bit weird, so I tried another way after the presentation. I tried to get the screenshot of the page and cut the part I want (which is the image I clicked). So this time I send the image to background to do style transfer. At first I thought doing style transfer in background won’t slow down the user doing things at front. However, it seems that it will slow down the whole browser. Fortunately, the image is not that large, so it only takes few seconds. However, when you hover to the image, there will come up with a div, and I cannot get rid of it, so the output image will have the div on it.


The full code is here: StyleTransfer

RWET Final Project: PhonoCi 2.0

From my previous homework, I made a poem structure called “PhonoCi”, which is a combination of Phonogram and Ci.

Ci is a type of poetry in Classical Chinese Poetry. Ci use a set of poetic meters derived from a base set of certain patterns, in fixed-rhythm, fixed-tone, and variable line-length formal types, or model examples.

For my homework last time, I just restricted the word counts and put English words into it. However, I found that it somehow lost the rhythm since some words are having too many syllables.

In Chinese, there are four tone classes for Chinese words, and for a certain Ci, it will give a certain pattern for poets to put words into it.


So this time I change to restrict the number of syllables. I restricted all my corpus in only one syllable, and few in two syllables.

Also, I try to “visualize” the poem so I go to Shutterstock and search for the images then combine them into a gif.

Some outputs:



The full code is here: PhonoCi 2.0

Hacking The Browser Final Project Thoughts

For the final project, I am going to do image style transfer extension which can apply on the images on the webpage, maybe Instagram in particular.
This project is combined with my Nature of Code project
The project can be separated to 3 big parts:
1. Train data for style model
2. Style Transfer the image
3. Swap the image

The first step would be done locally, since it will take a long time to train the data,
it is impossible for a user to randomly upload one image to be a style base image.

For the second step, I want to do it in background.js,
since the transfer process would last for some time according to the size of an image,
so putting behind the scene may ideally won’t stuck the user doing other things in the page.

For the last step, it is just showing the result on the page.

I think I would use a popup page for users to choose the image style they want to change,
also, I would like to have the user to choose which image they want to be changed according to mouse clicking,
so perhaps it will block the original behavior of the mouse clicking on the particular image component after they have chosen the style image.

So definitely I will use page action, content script, background script. For the APIs, right now I don’t have a particular list for it.

Hacking The Browser HW5

To simply describe how to bypass the WSJ work, just delete the cookie header and block the cookie being set when you receive the header. Also, add/modify Referer and User-Agent header to Google and a specific web browser name (OR in the update version, change it as you are from Twitter).

In manifest.json, it is simple that you need API permission of webRequest and webRequestBlocking for doing the request and blocking things. Then list out the websites that you want to work with.


For background.js, it will try to do something before the browser is sending the header and when the browser receives the header.


The first argument is a callback function, which is the action you want to do when the event is happening.

The second argument is filter, which limits the requests for which events are triggered – in this example, it it can work on all urls and the header type should be “name_frame”.

The third argument is opt_extraInfoSpec, which is used to decide what information is going to pass into the callback function – in this example, it sends requestHeaders and responseHeaders in these 2 events. For the blocking, it is used to tell the request is blocked until the callback function returns so that the header can have time to be modified.

In the changeRefer callback functions, it will return every header that is not Cookie, then check if the url you are requesting is being allowed or not, if not, delete the cookie. Then it will check if there are headers called Referer and User-Agent, if yes, change it to Google/Twitter and the web browser name.

In the blockCookies callback function, it will delete the header called Set-Cookie whenever the browser receives header

Detourning The Web Final Project

Most of the time, when I see a Hollywood movie English title, I always have no idea what it is. The main reason is that when a movie is released in Taiwan, the movie will have a Chinese name, and most of time, the Chinese name is not only translated from English.

So I want to grab the alternative titles in different countries and use google translate to translate back to English to see what’s different. Also, it is fun to see what google will translate from the foreign languages.

After doing some researches, I found that TMDB got API for users to get movie data. Accordingly, I make a page by FLASK for user to enter keywords to search for movies, then choose one movie will return posters with translated movie titles.

I use requests to get data from TMDB API, then use selenium to open a browser and do translation. The final step is to use pillow to put the title on a poster.


the full code is here: Movie Title Translation

Hacking The Browser HW3

For the first part, I put in github: Bookmaklet2Extension

For the second part, I chose a bookmarklet called “MapThis”, which can grab the address you select and directly redirect to google map. The code is showing below with comments on it.


For the third part, I chose an interesting chrome extension called Giphy Tabs, which will come up with a GIF every time you open a new tab.

NOC Final Project Proposal

For the final project, I am interested in working on machine learning concepts so I want to take this chance to make a machine learning / neural network project.

In Jabrils’ talk, he showed an image style transfer project:

Since I was working on some image processing projects and I am really interested in combining machine learning techniques into it.

Also, I found that there are some projects like Face Portraits Style Transfer , Style Transfer and Colorizing Grayscale.

So I want to take images and transfer them into a style of different kinds of paintings.

I don’t  know if it is too ambitious to do this or even makes sense to do this since there are already some developed projects or even mobile apps that can do these works.