Getting Started with Open Distro for ElasticSearch- 27 September - 15:00

Getting Started with Open Distro for ElasticSearch- 27 September - 15:00

• 1. Slide30 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Javier Ramirez @supercoco9 2019.09.28 Open Distro for Elasticsearch Istanbul loft :p
• 2. My personal journey into full-text search (and log processing) © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. My personal journey into full-text search (and log processing) Problem Solution SELECT LIKE % and SOUNDEX are not good enough Before 2001 The fuzzy years Say bye to transactions and hi to data corruption with MyISAM tables I need to support multi language, and speed, and to search by full text and facets, and rankings, column weights and excerpts. I want to be like Altavista Google 2001-2005 The Lucene epiphany Embrace JAVA Hello Lucene (and Nutch) I want full text search for dynamic languages Ruby FTW 2006-2009 The anything but Java golden age Sphinx Search * Spoke about Sphinx at several events My data-sets start growing to multiple servers I want to index anything, not just content from my database I would like to have the power of Lucene, without the pain of JAVA (python and JavaScript are cool), and the excess of configuration 2010-2013 The sol4r vs elastic civil wars Sol4r and Elastic are excellent choices to tame the raw power of Lucene and make It more approachable PostgreSQL good enough for many things. Great if you need GIS * Spoke about PostgreSQL full- text search at PGConf As a DevOps convert, apart from full-text search, I want operational search for the myriad logs of my microservices. I want to have pretty and powerful real-time dashboards, but either they are very SQL/Business-oriented or very low-level monitoring oriented. 2014-2018 The ELK take over With LogStash and Kibana Elasticsearch becomes not just my search engine, but also my log analytics engine. * Spoke about Logstash, then about ELK or ExK for log and big data analytics at several events
• 3. Slide1921 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source: TechCrunch survey of popular open source software from April’17 •Sometimes referred to as the “ELK Stack” – Elasticsearch, Logstash & Kibana •Distributed search and analytics engine built on Apache Lucene •Easy ingestion and visualization •Developed in Java What is Elasticsearch?
• 4. Machine data driving Elasticsearch growth © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Machine data driving Elasticsearch growth IT & DevOps: Databases, Servers, Storage, Networking Increase in IoT and Mobile Devices: Gaming, Sensors, Web Content Cloud-based architectures Machine-generated data is growing 10x faster than business data… Logs, logs, and more logs Source: insideBigData—The Exponential Growth of Data, February 16, 2017
• 5. Slide1925 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. •Lucene •Log4j •Joda •Protobuf •Disruptor •Jackson •XMLStreamWriter2 Standing on the shoulders of Open Source giants •Apache commons •Google Common •Netty •Apache httpcomponents •Apache Avalon •Maxmind db •Mustache •Logstash •Kibana •Doug Cutting •Lucene •Nutch •Hadoop •Shay Banon •Compass
• 6. How we think about OSS licensing and distribution? © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How we think about OSS licensing and distribution? Base OSS free of proprietary code Keep commercial software on top of OSS separate Allow anyone to innovate on OSS Don’t change licensing or distribution midstream
• 7. An Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise-grade security, alerting, SQL, and more © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. An Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise-grade security, alerting, SQL, and more
• 8. Benefits of Open Distro for Elasticsearch © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of Open Distro for Elasticsearch Enterprise-grade Delivering security and advanced capabilities such as alerting, SQL, and cluster diagnostics 100% open source Providing you the freedoms, so you can freely view, use, change, and distribute the code Community-driven Providing individuals and organizations the freedom to easily contribute changes to the distro
• 9. Flexible deployment options © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Flexible deployment options •Docker •RPM •Debian
• 10. Simple to get started © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simple to get started Visit the website Download the Elasticsearch and Kibana packages Load and query data 1 2 3
• 11. Open Distro for Elasticsearch - Features © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Open Distro for Elasticsearch - Features Security Alerting SQL Performance Analyzer Achieve encryption in-flight, fine-grained access control, audit logging, and compliance Monitor your data and send automatic alerts on any changes in your data Easily interact with your Elasticsearch cluster and extract insights using the familiar SQL query syntax Get deep visibility into system bottlenecks even when your Elasticsearch cluster is under duress.
• 12. SecurityKeep your data secure © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security Keep your data secure Encryption Keep your data secure when in transit Authentication Leverage your existing authentication infrastructure RBAC Granular access control over user actions on your cluster Audit logging Track and record all user actions and meet HIPAA, and PCI compliance
• 13. Access control flow for RBAC © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Request with credentials Access control flow for RBAC Authc — via basic HTTP auth, LDAP, AD, SAML, web tokens, SSL Authz — Backend identities mapped to Open Distro roles Permissions — allow a role to perform an action against a cluster/index/document/field Action groups — Groups of permissions Authc provider Authc Request with user/backend roles Roles and permissions Authz Response
• 14. Kibana multi-tenancy © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kibana multi-tenancy Group A Group B Group B permissions Group A permissions Index 1 Index 2
• 15. Encryption – Carl/Jon/Eli/Hardik © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Encryption – Carl/Jon/Eli/Hardik How strong Configurable TLS certs Configurable key-exchange algos Open SSL/Java SSL Audit log – file system (index to same or different cluster. Alert. Support compliance logging) – who accessed what. Succ/fail auth. Compliance someone changing security settings, cluster config – GDPR etc.
• 16. AlertingReceive alerts on your data © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alerting Receive alerts on your data Create monitors Query the data you want to and receive alerts on it Customize alert conditions Define alerting threshold and severity for multiple trigger conditions Get notifications Built-in integrations for webhook and Slack to get notified on the channels you use View alerts All alert executions are indexed for easy tracking and visualization
• 17. SQL Support Query data with SQL  © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SQL Support Query data with SQL Comprehensive SQL support Supports over 40 functions, data types, and commands including join support Translate SQL to JSON Create JSON using SQL to configure sophisticated access control policies Use existing tools Provides a JDBC driver so you can use a variety of business intelligence, analytics, and ETL tools
• 18. Performance AnalyzerGet deep diagnostic insights into your cluster © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance Analyzer Get deep diagnostic insights into your cluster Identify bottlenecks across the stack Provides a powerful REST API for querying Elasticsearch metrics to diagnose issues across stack Runs independent of your cluster Perform diagnostics even if the cluster is under duress Analyze hundreds of data points Supports over 60 metrics across 10 dimensions for instrumentation of your cluster health
• 19. PerfTop CLI © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PerfTop CLI •Provides pre-configured dashboards for analyzing cluster, node, and shard performance •Custom JSON templates to create the dashboards to diagnose your cluster performance
• 20. Mooooooarrrr plugins © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mooooooarrrr plugins •Index management (RFC open now) •jobs scheduler •What will you contribute? 
• 21. Community and Contributions © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Community and Contributions Open Distro for Elasticsearch’s success is driven by the community’s participation, contributions and innovation to the project. You can follow project discussions, engage with fellow community members, contribute PRs, file bugs or request a feature at: Discussion Forums https://discuss.opendistrocommunity.dev/ Community https://github.com/opendistro-for-elasticsearch/community/issues
• 22. Useful Links © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Useful Links Project Website and Technical Documentation https://opendistro.github.io/for-elasticsearch/ Source Code https://github.com/opendistro-for-elasticsearch
• 23. Thank you! © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Javier Ramirez @supercoco9
• 24. Slide1927 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Javier Ramirez @supercoco9 2019.09.27 Open Distro for Elasticsearch Search Engines Amsterdam