Friday, April 13, 2018

data cleaning

I saw a talk the other day by Anna Gilbert and it was pretty cool. It was about data cleaning and it was interesting!

I had never thought of data cleaning as interesting (or acceptable) because it always seemed like people were throwing out inconvenient data points willy-nilly. Like people would arbitrarily decide what an outlier was and just throw those data out. Horrifying.

Anna talked about the issue of getting noisy pairwise distance data. The thing about distance data is that it's supposed to come from a metric, but the noise can mess up the triangle inequality sometimes. Having a metric is usually useful downstream for other computations and can give better guarantees or results. So, in metric repair, the idea is to adjust some of the distances so that all the triangle inequalities are satisfied.

Anna's idea is to do sparse metric repair, which means changing as few distances as possible while making the distances satisfy the triangle inequality.

She didn't have an application. Someone pointed out that usually, all sensors have a little bit of noise so requiring sparsity isn't really necessary. My thoughts are that yes, sensors are noisy, but they usually give values close to ground truth. But sometimes sensors malfunction and give readings that are way off of ground truth. If you were trying to fix those extreme outliers, you would try to change as few distances as possible, but the ones you did change you might allow to be altered a significant amount. Possible applications are distributed sensing, robot swarms, and DNA sequencing for phylogenetic tree reconstruction.


  1. Very well written article. It was an awesome article to read. Complete rich content and fully informative. bond cleaning adelaide

  2. I read this article, it is really informative one. Your way of writing and making things clear is very impressive. Thanking you for such an informative article.Restaurant Hood Cleaning service in Malaysia

  3. commercial disinfecting, Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info.

  4. Nice post. I'm impressed by this blog. There is so much information about this topic. Thanks for sharing an amazing article like this. please keep posting. Once Off Cleaning Dublin

  5. You have done good work by publishing this article here. I found this article too much informative, and also it is beneficial to enhance our knowledge.Industrial cleaning company in newcastle Grateful to you for sharing an article like this.

  6. Couch cleaning You made such an interesting piece to read, giving every subject enlightenment for us to gain knowledge. Thanks for sharing the such information with us to read this...