Monitoring

Provide a step-by-step tutorial on how to instrument a Node.js application using the 'prom-client' library to expose custom metrics on the /metrics endpoint. Then, explain how to configure Prometheus to scrape these metrics every 15 seconds. Include example code for the Express server and the corresponding prometheus.yml configuration file.

Generate a Grafana dashboard JSON model that includes panels for CPU usage, Memory utilization, Network I/O, and Pod status across a Kubernetes cluster. Ensure the dashboard queries the standard kube-state-metrics and cadvisor data sources. Describe what each panel represents and suggest appropriate alert thresholds.

Write a Logstash pipeline configuration that accepts raw syslog input over port 5000. The configuration should use the grok filter to parse the timestamp, log level, source IP, and message. Finally, output the parsed data into an Elasticsearch index named 'syslog-%{+YYYY.MM.dd}'. Explain how each part of the configuration works.

Describe the key components required for an effective Application Performance Monitoring (APM) strategy in a microservices environment. Discuss the importance of Distributed Tracing (e.g., OpenTelemetry, Jaeger), Real User Monitoring (RUM), and database performance monitoring. List at least three popular open-source tools and three commercial tools suitable for this task.

Define a Prometheus recording rule to calculate the 95th percentile latency (p95) for an HTTP API endpoint named 'api_v1_request_duration_seconds'. Then, write an Alertmanager alert rule that fires a 'Warning' alert if the p95 latency exceeds 500ms for a duration of 5 minutes. Explain the syntax used for the histogram_quantile function and the 'for' clause.

Provide a configuration example for the Prometheus Blackbox Exporter. The configuration should include an HTTP probe that checks if 'https://example.com' is returning a 200 OK status code and if the response body contains the string 'Welcome'. Include the necessary scrape_config snippet for prometheus.yml to utilize this module.

Write a Python code snippet demonstrating how to initialize the OpenTelemetry SDK for a Flask application. The code should configure tracing to export spans to the console (standard output) and to Jaeger. Ensure the code includes automatic instrumentation for Flask requests and demonstrates creating a custom span for a specific function, such as a database query.

Draft a documentation guide outlining best practices for log retention and rotation. Address the following points: how to determine the appropriate retention period based on compliance requirements, strategies for compressing old logs, and how to implement hot/warm/cold storage tiers. Provide a sample logrotate configuration file for a generic application log stored at /var/log/app/app.log.

Write a detailed comparison between Synthetic Monitoring and Real User Monitoring (RUM). Create a table that highlights the differences in data collection method, use cases, advantages, and disadvantages. Provide specific scenarios where one approach would be preferred over the other, and explain how they complement each other in a full observability stack.

Explain the process of setting up monitoring to detect deadlocks in a PostgreSQL database. Identify the specific metrics and views (such as pg_stat_activity and pg_locks) that should be monitored. Provide a SQL query that can be used to extract details about current locks and waiting processes, and describe how to integrate this into a monitoring alerting system.