About the Role: We are creating an innovative synthetic data generation engine designed to produce highly realistic observability datasets, including metrics, logs, and traces, to enhance AI/ML training and benchmarking efforts. In this role, you will be responsible for designing, implementing, and scaling pipelines that simulate complex production environments while generating controllable and parameterized telemetry data.
What You Will Do:
• Design and implement generators for metrics, such as CPU usage, latency, and throughput, alongside logs in both structured and unstructured formats.
• Build flexible pipelines that enable control over data rate, shape, and the injection of anomalies.
• Develop repeatable workload simulations and system behaviors that encompass microservices, failures, and recoveries.
• Integrate synthetic data storage systems with tools like Prometheus, ClickHouse, or Elasticsearch.
• Collaborate closely with ML researchers to assess the realism and coverage of the generated datasets.
• Optimize systems for scalability and reproducibility utilizing Docker containers.
Who You Are:
• You possess strong programming skills in Python.
• You have familiarity with observability tools such as Grafana, Prometheus, ELK, and OpenTelemetry.
• You have a solid understanding of distributed systems, including metrics and log structures.
• You have experience in building data pipelines or synthetic data generators.
• (Bonus) Knowledge of anomaly detection, time-series analysis, or generative ML models is a plus.
Compensation: $50 - $75/hr based on experience.
Work Schedule: Remote, flexible hours.
Project Duration: 5-6 weeks.
