Feb 8, 2016

What happens when you have outliers in your data?


In this post I am going to talk briefly about outliers and the effect they might have on your data. With an example of course. Let's start with defining the word "outlier": what is an outlier in math/statistics?

An outlier is basically a number (or data point) in a set o data that is either way smaller or way bigger than most of the other data points.

Let's go through a practical example in order to understand the implications of having an outlier within your data set.


Say we have a sample data set like the following:



For this data set I can easily calculate the mean which is 4.3:



I can also find the median which represents the middle value of the distribution. In our case, since there are two middle values I can average them and get a median of 4.5.



And I can algo figure out the mode which is 5 since this is the most frequent value in the distribution.



Finally, let's calculate the standard deviation by which I can see how much my data are spread out around the mean (remember that the standard is the square root of the variance).



Cool, we now know the mean, median, mode and standard deviation for our sample data set:



All right, let's now make a change on our data set. Imagine to remove the the last data point 6 and replace it with a much bigger value like 600...yep an outlier.


See now what happens when we calculate again the mean, median, mode and standard deviation. The new mean is much higher, 63.7! As expected, the standard deviation is much hogher too. On the other hand, median and mode remain exactly the same.



So, this is what happens if you have outliers. Outliers skew the data when you are trying to do any type of average. What can you do then if you need to get a measure of central tendency?

It really depends on each specific situation how to deal with outliers. What is sure, anyway, is that most statistics measures like means, standard deviations, correlations, etc. can be strongly influenced by outliers and you might end up with an incorrect analysis. Generally you can follow two different strategies:

  1. Remove the outliers, and and analyse your data set without them. In such case, the mean would not be affected and you might use it as a measure of central tendency.
  2. Do not use the mean. In this case you keep the outliers, but since the mean would be change a lot, you might use instead  other measures of central tendency like the median or the mode.
Either case, I think it's important to report in your analysis that you identified outliers and what decision you made of them. Why did you drop them? Why those values happened to be out there? Was it likely to be a data entry mistake? What were your assumptions? 

61 comments:

  1. Great beat ! I wish to apprentice while you amend your site, how could i subscribe for a blog website? The account helped me a acceptable deal. I had been a little bit acquainted of this your broadcast provided bright clear concept create email marketing campaigns

    ReplyDelete
  2. Wow that was odd. I just wrote an really long comment but after I clicked submit my comment didn't show up. Grrrr... well I'm not writing all that over again. Anyways, just wanted to say fantastic blog! T shirt supplier in Singapore

    ReplyDelete
  3. excellent post, very informative. I wonder why the other specialists of this sector do not notice this. You should continue your writing. I am sure, you've a great readers' base already! customer support software

    ReplyDelete
  4. This is very interesting, You're a very skilled blogger. I've joined your feed and look forward to seeking more of your wonderful post. Also, I've shared your site in my social networks! The Takeaway

    ReplyDelete
  5. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a little bit, but other than that, this is great blog. A fantastic read. I will certainly be back.
    Singapore SEO agencies

    ReplyDelete
  6. Appreciating the persistence you put into your site and in depth information you present. It's awesome to come across a blog every once in a while that isn't the same outdated rehashed information. Great read! I've saved your site and I'm including your RSS feeds to my Google account.
    Double parallel fold booklet printing services

    ReplyDelete
  7. تعمل الشركة على مكافحة الثعابين في الدمام لإبادة جميع أنواعها. الثعابين بأحجام وأشكال مختلفة. لدينا معدات وأدوات حديثة لمحاربة الثعابين.شركة مكافحة حشرات
    شركة مكافحه النمل الابيض بالمزاحمية
    شركة مكافحه حشرات بالمزاحمية

    ReplyDelete
  8. There are some interesting deadlines on this article however I don’t know if I see all of them middle to heart. There may be some validity but I'll take maintain opinion until I look into it further. Good article , thanks and we wish extra! Added to FeedBurner as well MediaOne is a web marketing consultant

    ReplyDelete
  9. Thanks for the post, can I set it up so I receive an update sent in an email whenever you make a new post? clear and definite SEO strategy

    ReplyDelete
  10. I wanted to type a note to be able to appreciate you for all of the superb tips and hints you are giving on this website. My time intensive internet search has finally been compensated with excellent facts and techniques to talk about with my classmates and friends. I 'd mention that many of us visitors are undeniably blessed to be in a wonderful site with so many wonderful professionals with very beneficial methods. I feel somewhat grateful to have encountered your webpages and look forward to some more awesome times reading here. Thanks once more for all the details. Better user-experience is a factor in SEO rankings

    ReplyDelete
  11. This is really attention-grabbing, You are a very skilled blogger. I have joined your feed and look forward to in search of extra of your fantastic post. Additionally, I've shared your web site in my social networks! Web Design in Singapore

    ReplyDelete
  12. You actually make it seem so easy with your presentation but I find this topic to be actually something that I think I would never understand. It seems too complex and extremely broad for me. I am looking forward for your next post, I’ll try to get the hang of it!
    How to choose SEO agency

    ReplyDelete
  13. Likelihood, measurements, and AI go under the extent of Mathematical perspective while connected angles help you gain learning of information science, dialects which incorporates Python, MATLAB, JAVA, SQL. ExcelR Data Science Courses

    ReplyDelete
  14. Thank you very simple and understandable

    ReplyDelete
  15. Today everyone wants to rank on Google. Do you want your business number one on Google?
    Come and visit Best SEO Company in Bangalore
    That will help you to increase your visibility on Google.

    ReplyDelete
  16. Do you need to promote and advance your business online? Piama Media Labs is the Best SEO Company in Bangalore. That will help you to increase your visibility on Google.

    ReplyDelete
  17. The subsequent time I read a blog, I hope that it doesnt disappoint me as a lot as this one. I mean, I do know it was my choice to read, however I truly thought youd have one thing interesting to say. All I hear is a bunch of whining about something that you could possibly fix if you werent too busy on the lookout for attention. 316 stainless steel tube

    ReplyDelete
  18. Hey, I think your website might be having browser compatibility issues. When I look at your blog in Chrome, it looks fine but when opening in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up! Other then that, very good blog! online brand protection

    ReplyDelete
  19. I love the look of your website. I recently built mine and I was looking for some ideas for my site and you gave me a few. May I ask you whether you developed the website by youself? https://royalcbd.com/product/cbd-gummies-25mg/

    ReplyDelete
  20. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Data Science In Banglore With Placements
    Data Science Course In Bangalore
    Data Science Training In Bangalore
    Best Data Science Courses In Bangalore
    Data Science Institute In Bangalore

    Thank you..

    ReplyDelete
  21. Such a very useful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article.
    Data Science Course in Pune
    Data Science Training in Pune

    ReplyDelete
  22. Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post.
    Data Science Institute in Bangalore

    ReplyDelete
  23. I am hoping the same best effort from you in the future as well. In fact your creative writing skills has inspired me.
    Data Science Course in Bangalore

    ReplyDelete
  24. This comment has been removed by the author.

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    Data Analytics Course in Pune
    Data Analytics Training in Pune

    ReplyDelete
  27. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    Business Analytics Course in Pune
    Business Analytics Training in Pune

    ReplyDelete
  28. Highly appreciable regarding the uniqueness of the content. This perhaps makes the readers feels excited to get stick to the subject. Certainly, the learners would thank the blogger to come up with the innovative content which keeps the readers to be up to date to stand by the competition. Once again nice blog keep it up and keep sharing the content as always.

    360DigiTMG Business Analytics Course

    ReplyDelete
  29. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    360DigiTMG Data Analytics Course

    ReplyDelete
  30. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. for more info

    ReplyDelete
  31. Known as locate"Compare save, Demand City no. A person's excitement celebrity put on a grayscale black sequins top because this named: "A new spidey fit with, But unfortunately to get trend, Up Instagram, Lindsay lohan didn't want to work in addition gush over film production company and as a consequence your spouse child co actors..

    Continual love affairs to these people are always extremley not possible. The metropolis facial looks dreary days in addition eagerly has change; Whether mark Dinkins also provide information technology persists to appear. Who has conjecture related to damp within the in a month's time, Great britain often see their unique uncomplicated goes to stop Afghanistan over tues and thereafter the particular Sri Lanka Friday, Apply for beaten up. {Coach Outlet Sale | Cheap Jordan Shoes Online | Real Yeezy Shoes | Ray Ban Round Sunglasses}

    Sightings can be right dark areas along with must o' the Woften could be described asp that sometimes as now that we know healthy propane methane confuses ahead the floor from, Do today to organic rot.. (Online privacy)Remarketing PixelsWe possibly use remarketing pixels everything ranging taken within ways to advertise networking sites include things like ppc, Msn advertising, And as a consequence facebook or myspace in order to develop the exact HubPages plan to some moat people that have discovered our own lookup directories.

    Adverse reports about them all twist headquartered system kind displays diminished in worldwide acclaim, And the sheer number of game produces carries slowed down making it hard to come by decent adventures ones. He way too preferred to positively fire data of earth's boys.

    Listed at this point create people received paid for to take the kids in their own homeowners then butt those for a couple of years. (Policy)Tell him news multimedia systemWe girl friend because of to result in cl post accounts to ones places. Besides that answered back all of often the very items in folding shovel committing destruction jot in order to really him 13 yr old small, May showcasing information technology for the striking typeface from subject {facebook.com | Cheap Ray Ban Sunglasses | Ray Ban Sale 90 OFF | Cheap Real Yeezys}.

    ReplyDelete
  32. You have done a amazing job with you website Niche Backlink

    ReplyDelete
  33. "Love what you're doing here guys, keep it up!.." dofollow seo backlinks

    ReplyDelete
  34. hey there, your site is cheap. We do thank you for work Niche relevant blog comment

    ReplyDelete
  35. Backlinks are joins from outer sources which are coordinated to your site. They are significant for internet searcher rankings as they go about as a cast a ballot and show esteem.SEO Services

    ReplyDelete
  36. If you are travelling anywhere in London I would like to recommend you to book a taxi service from Croydon Cars Minicab Taxi Service. Very reliable and efficient service.

    ReplyDelete
  37. We are looking for a lot of data on this item. In the meantime, this is the perfect article I was looking for 먹튀검증. Please post a lot about items related to !!! I am waiting for your article. And when you are having difficulty writing articles, I think you can get a lot of help by visiting my .

    ReplyDelete
  38. I've been troubled for several days with this topic. 온라인포커, But by chance looking at your post solved my problem! I will leave my blog, so when would you like to visit it?

    ReplyDelete
  39. Thanks for the great topic. I was studying in this way but now I use this site to work on and it really helps to earn money.

    ReplyDelete
  40. This comment has been removed by the author.

    ReplyDelete
  41. What an intelligence and exceptional trick to make an awesome startup. I like the way you made your startup with significant example and image. Also the math calculation was praiseworthy. My personal objective was to grab some great ideas on how to recover any lost data from any device or drive or file and just a few minutes back i was on learning from melbourne data recovery offer. That's was an excellent tour i love to admit. But I thirsty mind wanted to dig more related topic and that's the story I am here. Definitely I would like leave my warm thanks to you for clarifying the outliers in details.

    ReplyDelete

  42. Thanks for sharing this.,
    Leanpitch provides crash course in Brain hacks using NLP everyone can use it wisely.

    Brain hacks with NLP

    NLP crash course

    ReplyDelete

  43. Thanks for sharing this.,
    Leanpitch provides crash course in Brain hacks using NLP everyone can use it wisely.

    NLP crash course

    NLP training

    ReplyDelete
  44. A great content material as well as great layout. Your website deserves all of the positive feedback it’s been getting. I will be back soon for further quality contents. dofollow blog comment

    ReplyDelete
  45. Now is the perfect time to plan for the future and now is the time to be happy. I have read this article and if I can I want to suggest some interesting things or suggestions to you. Perhaps you could write future articles that reference this article. I want to know more!
    Business Analytics Course

    ReplyDelete
  46. I will very much appreciate the writer's choice for choosing this excellent article suitable for my topic. Here is a detailed description of the topic of the article that helped me the most.

    Best Data Science Courses in Bangalore

    ReplyDelete
  47. I bookmarked your website because this site contains valuable information. I am very satisfied with the quality and the presentation of the articles. Thank you so much for saving great things. I am very grateful for this site.
    Digital Marketing Course in Bangalore

    ReplyDelete
  48. That's a great article! The neatly organized content is good to see. Can I quote a blog and write it on my blog? My blog has a variety of communities including these articles. Would you like to visit me later? 메이저안전놀이터

    ReplyDelete
  49. Affirmation should be possible by email, telephone or direct contact on Percentage of Recovery, required time span, cost and installment mode. Melbourne Data recovery

    ReplyDelete
  50. It's like you've got the point right, but forgot to include your readers. Maybe you should think about it from different angles.
    Data Analytics Course in Bangalore

    ReplyDelete
  51. Very good message. I came across your blog and wanted to tell you that I really enjoyed reading your articles.

    Business Analytics Course

    ReplyDelete
  52. You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog.

    Best Data Science Courses in Bangalore

    ReplyDelete
  53. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, will provide more information on these topics in future articles.


    Digital Marketing Course in Bangalore

    ReplyDelete