50 shades of ES

What does it take to setup a 40+ node ES cluster able to handle tens of thousands of insertions while serving thousands of search queries?In this presentation I will try to guide you through our expierence at Infobip and what we:- failed at,- wanted to know before failing and- learned on the way.Our concrete use case will presented together with all its aspects from hardware consideration to client service implementation.The use case is simple, provide a way to query half a year of business logs to clients and internal support looking for transaction logs, clients wanting aggregated reports and business analysts wanting to indulge in the data in indicent ways.Implementation considerations are many:Do we need one or more clusters? How many machines? Which ones? Where to put them? How to organise indices?Who or what is going to manage the indices?Do we need some kind of API gateway for the clients?Where to store the indices configuraitons?How to test the cluster?How to fill the cluster?How to migrate the data?What to do when ingestion/search rate is slow?Cluster is down, why o'why?How to join kafka streams in ES?Using Ansible for managing the cluster.Cluster and dataflow monitoring stack.And many more...