Google Git
Sign in
edge / apidAnalytics / refs/heads/apid-build
commitaf7d55751868ffd4b081cc06ad5dbd6d569ca7e7[log] [tgz]
authorScott Ganyo <scott@ganyo.com>Mon Feb 13 18:28:35 2017 -0800
committerScott Ganyo <scott@ganyo.com>Mon Feb 13 18:28:35 2017 -0800
tree83b76906f7908fac4e057e9243a563d1734c4dae
parent2ff4236780f849c44f9fa1e7094ad961dc8ad58d [diff]
Refactor apid -> apid-core
5 files changed
tree: 83b76906f7908fac4e057e9243a563d1734c4dae
  1. api.go
  2. api.yaml
  3. api_helper.go
  4. api_helper_test.go
  5. api_test.go
  6. apidAnalytics_suite_test.go
  7. buffering_manager.go
  8. buffering_manager_test.go
  9. common_helper.go
  10. common_helper_test.go
  11. crash_recovery.go
  12. glide.yaml
  13. init.go
  14. init_test.go
  15. listener.go
  16. listener_test.go
  17. pluginData.go
  18. README.md
  19. upload_manager.go
  20. upload_manager_test.go
  21. uploader.go
  22. uploader_test.go
README.md

ApidAnalytics

This is a core plugin for apid and is responsible for collecting analytics data for runtime traffic from Micro and Enterprise Gateway and puplishing to Apigee.

Configuration

namedescription
apidanalytics_base_pathstring. default: /analytics
apidanalytics_data_pathstring. default: /ax
apidanalytics_collection_intervalint. seconds. default: 120
apidanalytics_upload_intervalint. seconds. default: 5
apidanalytics_uap_server_basestring. url. required.
apidanalytics_use_cachingboolean. default: true
apidanalytics_buffer_channel_sizeint. number of slots. default: 100

Startup Procedure

  1. Initialize crash recovery, upload and buffering manager to handle buffering analytics messages to files locally and then periodically upload these files to S3/GCS based on signedURL received from uapCollectionEndpoint exposed via edgex proxy
  2. Create a listener for Apigee-Sync event
    1. Each time a Snapshot is received, create an in-memory cache for data scope and developer information
    2. Each time a changeList is received
      1. if data_scope info changed, then insert/delete info for changed scope from tenantCache
      2. if any other kms table is changed, then refresh entire developerInfo cache as only specific fields are saved in the cache
  3. Initialize POST /analytics/{scope_uuid} API
  4. Upon receiving POST requests
    1. Validate and enrich each batch of analytics records
    2. If valid, then publish records to an internal buffer channel
  5. Buffering Logic
    1. Buffering manager creates listener on the internal buffer channel and thus consumes messages as soon as they are put on the channel
    2. Based on the current timestamp either an existing directory is used to save these messages or new a new timestamp directory is created
    3. If a new directory is created, then an event will be published on the closeBucketEvent Channel at the expected directory closing time
    4. The messages are stored in a file under tmp/<timestamp_directory>
    5. Based on collection interval, periodically the files in tmp are closed by the routine listening on the closeBucketEvent channel and the directory is moved to staging directory
  6. Upload Manager
    1. The upload manager periodically checks the staging directory to look for new folders
    2. When a new folder arrives here, it means all files under that are closed and ready to uploaded
    3. Tenant info is extracted from the directory name and the files are sequentially uploaded to S3/GCS
    4. Based on the upload status
      1. If upload is successful then directory is deleted from staging and previously failed uploads are retried
      2. if upload fails, then upload is retried 3 times before moving the directory to failed directory
  7. Crash Recovery is a one time activity performed when the plugin is started to cleanly handle open files from a previous Apid stop or crash event

Exposed API

POST /analytics/{bundle_scope_uuid}

Complete spec is listed in api.yaml