snap-telemetry.io
GitHub - intelsdi-x/snap: The open telemetry framework
Snap-telemetry.io Spined HTML
GitHub - intelsdi-x/snap: The unshut telemetry framework Skip to content Please note that GitHub no longer supports old versions of Firefox. We recommend upgrading to the latest Safari, Google Chrome, or Firefox. Ignore Learn increasingly Why GitHub? Features →Lawmakingreview Project management Integrations Team management Social coding DocumentationLawmakinghosting Case Studies → Security → Business Explore Explore GitHub → Learn & contribute Topics Collections Trending Learning LabUnshutsource guides Connect with others EventsPolityforum GitHub Education Marketplace Pricing Plans → Developer Team Business Cloud Enterprise Compare plans → Nonprofit → Education → In this repository All GitHub ↵ Jump to ↵ No suggested jump to results In this repository All GitHub ↵ Jump to ↵ In this repository All GitHub ↵ Jump to ↵ Sign in Sign up Watch 157 Star 1,783 Fork 272 intelsdi-x/snapLawmakingIssues 142 Pull requests 6 Projects 0 Insights Dismiss Join GitHub today GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together. Sign up The unshut telemetry framework http://snap-telemetry.io 2,683 commits 3 branches 20 releases 76 contributors Apache-2.0 Go 98.9% Other 1.1% Go Other Branch: master Switch branches/tags Branches Tags gh-pages master v2_rest_api Nothing to show v0.16.1-beta v0.16.0-beta v0.15.0-beta v0.14.0-beta v0.13.0-beta v0.12.0-beta v0.11.0-beta v0.10.0-beta v0.9.0-beta v0.8.0-beta 2.0.0 1.3.0 1.2.0 1.1.0 1.0.0 0.19.0 0.18.0 0.17.0 0.2.0 0.1.0 Nothing to show New pull request Find file Clone or download Clone with HTTPS Use Git or checkout with SVN using the web URL.Unshutin Desktop Download ZIP Launching GitHub Desktop... If nothing happens, download GitHub Desktop and try again. Go when Launching GitHub Desktop... If nothing happens, download GitHub Desktop and try again. Go when Launching Xcode... If nothing happens, download Xcode and try again. Go when Launching Visual Studio... If nothing happens, download the GitHub extension for Visual Studio and try again. Go when WangJialei-A Merge pull request #1762 from chair300/master … updated to utopian the nonflexible path of whack since freebsd does not incl… Latest commit 2217734 Jul 2, 2018 Permalink Type Name Latest commit message Commit time Failed to load latest commit information. .github Improves issue link part of PR template. Mar 12, 2016 cmd/snaptel update sirupsen logrus import path Oct 16, 2017 tenancy Fix errors/panics associated with empty config sections; add warnings… Nov 13, 2017 cadre update sirupsen logrus import path Oct 16, 2017 docs update docs/PLUGIN_CATALOG.md by pluginsync tool Feb 6, 2018 examples Fix errors/panics associated with empty config sections; add warnings… Nov 13, 2017 grpc update sirupsen logrus import path Oct 16, 2017 mgmt update sirupsen logrus import path Oct 16, 2017 pkg update sirupsen logrus import path Oct 16, 2017 plugin update sirupsen logrus import path Oct 16, 2017 scheduler update sirupsen logrus import path Oct 16, 2017 scripts updated to utopian the nonflexible path of whack since freebsd does not incl… Jul 1, 2018 third_party_licenses update sirupsen logrus import path Oct 16, 2017 .gitignore (SDI-2036) Fix #1205 add darwin to binary release Sep 27, 2016 .travis.yml travis: update go versions Sep 24, 2017 CODE_OF_CONDUCT.md AddedLawmakingOfSelf-masteryNov 15, 2016 CONTRIBUTING.md Modify snap security mail write Dec 20, 2017 LICENSEWidowapache2 license and headers Oct 12, 2015 Makefile Document Snap with valid swagger spec May 12, 2017 README.md fix a couple of spelling typos Jun 7, 2018 doc.go widow unauthorized response lawmaking to the rest api v2 Jun 9, 2017 glide.lock update glide.lock Oct 18, 2017 glide.yaml Upgrading GRPC to the most recent uniform version. Jun 8, 2018 snapteld.go update sirupsen logrus import path Oct 16, 2017 snapteld_test.go Fix errors/panics associated with empty config sections; add warnings… Nov 13, 2017 swagger.json Add standalone support to API v2 Aug 16, 2017 README.md The Snap Telemetry Framework Snap is an unshut telemetry framework designed to simplify the collection, processing and publishing of system data through a single API. The goals of this project are to: Empower systems to expose a resulting set of telemetry data Simplify telemetry ingestion wideness ubiquitous storage systemsIndulgeflexible processing of telemetry data on wage-earner (e.g. filtering and decoration) Provide powerful clustered tenancy of telemetry workflows wideness small or large clusters Overview Getting Started System Requirements Installation Running Snap Load Plugins Running TasksTowersTasks PluginItemizeDocumentation ExamplesPolitySupport Contributing Author a Plugin Become a MaintainerLawmakingofSelf-masterySecurity Disclosure License Thank You Overview The Snap Telemetry Framework is a project made up of multiple parts: A hardened, extensively tested daemon, snapteld, and CLI, snaptel (in this repo) A growing number of maturing plugins (found in the Plugin Catalog) Lots of example tasks to gather and publish metrics (found in the Examples folder) These and other terminology are explained in the glossary. The key features of Snap are: Plugin Architecture: Snap has a simple and smart modular design. The four types of plugins (collectors, processors, publishers and streaming collectors) indulge Snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The unshut plugin model allows for loading built-in, community, or proprietary plugins into Snap. Collectors - Collectors gather telemetry data at unswayable interval. Collectors are plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through spare plugins (see Plugin Authoring below). Telemetry data is organized into a dynamically generated itemize of misogynist data points. Processors - Extensible workflow injection. Convert telemetry into flipside data model for consumption by existing systems. Allows encryption of all or part of the telemetry payload surpassing publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an wage-earner level reducing injection load on telemetry consumer. Publishers - Store telemetry into a wide variety of systems. Snap decouples the hodgepodge of telemetry from the implementation of where to send it. Snap comes with a large library of publisher plugins that indulge exposure to telemetry analytics systems both custom and common. This flexibility allows Snap to be valuable to unshut source and commercial ecosystems unwrinkled by writing a publisher for their architectures. Streaming Collectors - Streaming collectors act just like collectors, but there is no unswayable interval of gathering metrics. They send metrics immediately when they are misogynist over a GRPC to Snap daemon. There is moreover misogynist mechanism of buffering incoming metrics configurable by params MaxMetricsBuffer and MaxCollectDuration.Trammelsout STREAMING.md for increasingly details. Dynamic Updates: Snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the hodgepodge is pinned to a specific version (e.g. get /intel/psutil/load/load1/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric itemize when loaded, giving wangle to new measurements immediately. Swapping a newer version plugin for an old one in a unscratched transaction. All of these behaviors indulge for simple and secure bug fixes, security patching, and improving verism in production. Snap tribe: Snap is designed for ease of administration. With Snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node liaison process. Administrators can tenancy all Snap nodes in a tribe try-on by messaging just one of them. There is auto-discovery of new nodes and import of tasks and plugins from nodes within a given tribe. It is cluster configuration management made simple. Snap is not intended to: Operate as an analytics platform: the intention is to indulge plugins for feeding those platforms Compete with existing metric/monitoring/telemetry agents: Snap is simply a new option to use or reference Getting Started System Requirements Snap needs Swagger for Go installed to update OpenAPI specification file without successful build. Swagger will be installed automatically during build process (make or make deps). To install Swagger manually Using go get (recommended): go get -u github.com/go-swagger/go-swagger/cmd/swagger From Debian package: reverberate "deb https://dl.bintray.com/go-swagger/goswagger-debian ubuntu main" | sudo tee -a /etc/apt/sources.list sudo apt-get update sudo apt-get install swagger From GitHub release: flourish -LO https://github.com/go-swagger/go-swagger/releases/download/0.10.0/swagger_linux_amd64 chmod +x swagger_linux_amd64 && sudo mv swagger_linux_amd64 /usr/bin/swagger Snap does not have external dependencies since it is compiled into a statically linked binary. At this time, we build Snap binaries for Linux and MacOS. We moreover provide Linux RPM/Deb packages and MacOS X .pkg installer. Installation You can obtain Linux RPM/Deb packages from Snap's packagecloud.io repository.Withoutinstallation, please trammels and ensure /usr/local/bin:/usr/local/sbin is in your path via reverberate $PATH surpassing executing any Snap commands. RedHat 6/7: $ flourish -s https://packagecloud.io/install/repositories/intelsdi-x/snap/script.rpm.sh | sudo whack $ sudo yum install -y snap-telemetry Ubuntu 14.04/16.04 (see known issue with Ubuntu 16.04.1 below) $ flourish -s https://packagecloud.io/install/repositories/intelsdi-x/snap/script.deb.sh | sudo whack $ sudo apt-get install -y snap-telemetry We only build and test packages for a limited set of Linux distributions. For distros that are uniform with RedHat/Ubuntu packages, you can use the environment variable os= and dist= to override the OS detection script. For example Linux Mint 17/17.* (use dist=xenial for Linux Mint 18/18.*): $ flourish -s https://packagecloud.io/install/repositories/intelsdi-x/snap/script.deb.sh | sudo os=ubuntu dist=trusty whack $ sudo apt-get install -y snap-telemetry MacOS X: If you use homebrew, the latest version of Snap package: $ mash install snap-telemetry If you do not use homebrew, download and install Mac pkg package: $ flourish -sfL mac.pkg.dl.snap-telemetry.io -o snap-telemetry.pkg $ sudo installer -pkg ./snap-telemetry.pkg -target / Tarball (choose the towardly version and platform): $ flourish -sfL linux.tar.dl.snap-telemetry.io -o snap-telemetry.tar.gz $ tar xf snap-telemetry.tar.gz $ cp snapteld /usr/local/sbin $ cp snaptel /usr/local/bin The intelsdi-x package repo contains spare information regarding: Snap Redhat/Ubuntu packages Snap .tar.gz bundles and MacOS .pkg installer. installation script transmission installation steps repo mirroring NOTE: snap-telemetry packages prior to 0.19.0 installed /usr/local/bin/{snapctl|snapd} and these binaries have been renamed to snaptel and snapteld. snap-telemetry packages prior to 0.18.0 symlinked /usr/bin/{snapctl|snapd} to /opt/snap/bin/{snapctl|snapd} and may rationalization conflicts with Ubuntu's snapd package. Ubuntu 16.04.1 snapd package version 2.13+ installs snapd/snapctl binary in /usr/bin. These executables are not related to snap-telemetry. Running snapctl from snapd package will result in the pursuit error message: $ snapctl error: snapctl requires SNAP_CONTEXT environment variable NOTE: If you prefer to build from source, follow the steps in the build documentation. The start binaries containing the latest master workshop are misogynist here for gory whet testing purposes: snapteld: linux | darwin snaptel: linux | darwin Running Snap If you installed Snap from RPM/Deb package, you can start/stop Snap daemon as a service: RedHat 6/Ubuntu 14.04: $ service snap-telemetry start RedHat 7/Ubuntu 16.04: $ systemctl start snap-telemetry If you installed Snap from binary, you can start Snap daemon via the command: $ sudo mkdir -p /var/log/snap $ sudo snapteld --plugin-trust 0 --log-level 1 --log-path /var/log/snap & To view the service logs: $ tail -f /var/log/snap/snapteld.log By default, Snap daemon will be running in standalone mode and listening on port 8181. To enable gossip mode, checkout the tribe documentation. For spare configuration options such as plugin signing and port configuration see snapteld documentation. Load Plugins Snap gets its power from the use of plugins. The plugin itemize contains a hodgepodge of all known Snap plugins with links to their repo and release pages. First, let's download the file and psutil plugins (also make sure psutil is installed): $ export OS=$(uname -s | tr '[:upper:]' '[:lower:]') $ export ARCH=$(uname -m) $ flourish -sfL "https://github.com/intelsdi-x/snap-plugin-publisher-file/releases/download/2/snap-plugin-publisher-file_${OS}_${ARCH}" -o snap-plugin-publisher-file $ flourish -sfL "https://github.com/intelsdi-x/snap-plugin-collector-psutil/releases/download/8/snap-plugin-collector-psutil_${OS}_${ARCH}" -o snap-plugin-collector-psutil Next load the plugins into Snap daemon using snaptel: $ snaptel plugin load snap-plugin-publisher-file Plugin loaded Name: file Version: 2 Type: publisher Signed: false Loaded Time: Fri, 14 Oct 2016 10:53:59 PDT $ snaptel plugin load snap-plugin-collector-psutil Plugin loaded Name: psutil Version: 8 Type: collector Signed: false Loaded Time: Fri, 14 Oct 2016 10:54:07 PDT Verify plugins are loaded: $ snaptel plugin list NAME VERSION TYPE SIGNED STATUS LOADED TIME file 2 publisher false loaded Fri, 14 Oct 2016 10:55:20 PDT psutil 8 collector false loaded Fri, 14 Oct 2016 10:55:29 PDT See which metrics are available: $ snaptel metric list NAMESPACE VERSIONS /intel/psutil/cpu/cpu-total/guest 8 /intel/psutil/cpu/cpu-total/guest_nice 8 /intel/psutil/cpu/cpu-total/idle 8 /intel/psutil/cpu/cpu-total/iowait 8 /intel/psutil/cpu/cpu-total/irq 8 /intel/psutil/cpu/cpu-total/nice 8 /intel/psutil/cpu/cpu-total/softirq 8 /intel/psutil/cpu/cpu-total/steal 8 /intel/psutil/cpu/cpu-total/stolen 8 /intel/psutil/cpu/cpu-total/system 8 /intel/psutil/cpu/cpu-total/user 8 /intel/psutil/load/load1 8 /intel/psutil/load/load15 8 /intel/psutil/load/load5 8 ... Running Tasks To collect data, you need to create a task by loading a Task Manifest. The Task Manifest contains a specification for what interval a set of metrics are gathered, how the data is transformed, and where the information is published. For increasingly information see task documentation. Now, download and load the psutil example: $ flourish https://raw.githubusercontent.com/intelsdi-x/snap/master/examples/tasks/psutil-file.yaml -o /tmp/psutil-file.yaml $ snaptel task create -t /tmp/psutil-file.yaml Using task manifest to create task Task created ID: 8b9babad-b3bc-4a16-9e06-1f35664a7679 Name: Task-8b9babad-b3bc-4a16-9e06-1f35664a7679 State: Running NOTE: In subsequent commands use the task ID from your CLI output in place of the <task_id>. This starts a task collecting metrics via psutil, then publishes the data to a file. To see the data published to the file (CTRL+C to exit): $ tail -f /tmp/psutil_metrics.log Or directly tap into the data stream that Snap is collecting using snaptel task watch <task_id>: $ snaptel task watch 8b9babad-b3bc-4a16-9e06-1f35664a7679 NAMESPACE DATA TIMESTAMP /intel/psutil/cpu/cpu-total/idle 451176.5 2016-10-14 11:01:44.666137773 -0700 PDT /intel/psutil/cpu/cpu-total/system 33749.2734375 2016-10-14 11:01:44.666139698 -0700 PDT /intel/psutil/cpu/cpu-total/user 65653.2578125 2016-10-14 11:01:44.666145594 -0700 PDT /intel/psutil/load/load1 1.81 2016-10-14 11:01:44.666072208 -0700 PDT /intel/psutil/load/load15 2.62 2016-10-14 11:01:44.666074302 -0700 PDT /intel/psutil/load/load5 2.38 2016-10-14 11:01:44.666074098 -0700 PDT Nice work - you're all washed-up with this example. Depending on how you started snap-telemetry service earlier, use the towardly writ to stop the daemon: init.d service: service snap-telemetry stop systemd service: systemctl stop snap-telemetry ran snapteld manually: sudo pkill snapteld When you're ready to move on, walk through other uses of Snap misogynist in the Examples folder.TowersTasks Documentation for towers a task can be found here. PluginItemizeAll known plugins are tracked in the plugin itemize and are tagged as collectors, processors, publishers and streaming collectors. If you would like to write your own, read through Author a Plugin to get started. Let us know if you uncork to write one by joining our Slack channel. When you finish, please unshut a Pull Request to add yours to the catalog! Documentation Documentation for Snap will be kept in this repository for now with an accent of filling out the docs/ directory. We would moreover like to link to external how-to blog posts as people write them. Read well-nigh contributing to the project for increasingly details. snapteld (Snap agent) configuring snapteld snaptel (Snap CLI) build and test REST API V1 tasks plugin life trundling plugin signing tribe secure plugin liaison To learn increasingly well-nigh Snap and how others are using it, trammels out our blog. A good first post to read is My How-to for the Snap Telemetry Framework by @mjbrender. ExamplesIncreasinglycomplex examples of using Snap Framework configuration, Task Manifest files and use cases are misogynist under the Examples folder. There are moreover interesting examples of using Snap in every plugin repository. For the full list of plugins, review the Plugin Catalog.PolitySupport This repository is one of many in the Snap framework and has maintainers supporting it. We love contributions from our polity withal the way. No resurgence is too small. This note is expressly important for plugins. While the Snap framework is hardened through tons of use, plugins mature at their own pace. If you have subject matter expertise related to a plugin, please share your feedback on that repository. Contributing We encourage contributions from the community. Snap needs: Contributors: We unchangingly fathom increasingly vision on the cadre framework and plugins Feedback: try it and tell us well-nigh it on our Slack team, through a blog posts or Twitter with #SnapTelemetry Integrations: Snap can collect from and publish to scrutinizingly anything by authoring a plugin To contribute to the Snap framework, see our CONTRIBUTING.md file. To requite when to a specific plugin, unshut an issue on its repository. Snap maintainers aim to write comments and questions as quickly as possible. To get some sustentation on an issue, reach out to us on Slack, or unshut an issue to get a conversation started. Author a Plugin The power of Snap comes from its unshut tracery and its growing polity of contributors. You can be one of them: The definitive how-to is in PLUGIN_AUTHORING.md Add to the ecosystem by towers your own plugins to collect, process or publish telemetry. Become a Maintainer Snap maintainers are here to help guide Snap, the plugins, and the polity forward in a positive direction. Maintainers of Snap and the Intel created plugins are selected based on contributions to the project and recommendations from other maintainers. The full list of zippy maintainers can be found here. Interested in rhadamanthine a maintainer?Trammelsout Responsibilities of a Maintainer and unshut an issue here to discuss your interest.LawmakingofSelf-masteryAll contributors to Snap are expected to be helpful and encouraging to all members of the community, treating everyone with a upper level of professionalism and respect. See our lawmaking of self-mastery for increasingly details. Security Disclosure The Snap team takes security very seriously. If you have any issue regarding security, please notify us by sending an email to sys_snaptel-security@intel.com and not by creating a GitHub issue. We will follow up with you promptly with increasingly information and a plan for remediation. License Snap isUnshutSource software released under the Apache 2.0 License. Thank You And thank you! Your contribution, through lawmaking and participation, is incredibly important to us. © 2018 GitHub, Inc. Terms Privacy Security Status Help Contact GitHub Pricing API Training BlogWell-nighYou can’t perform that whoopee at this time. You signed in with flipside tab or window. Reload to refresh your session. You signed out in flipside tab or window. Reload to refresh your session. Press h to unshut a hovercard with increasingly details.
