Why not both, dump to S3 and write pointers to kafka for portable event-based in... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		prpl on Oct 22, 2023 \| parent \| context \| favorite \| on: OpenTelemetry at Scale: Using Kafka to handle burs... Why not both, dump to S3 and write pointers to kafka for portable event-based ingestion (since everybody does messages a bit differently)

bushbaba on Oct 22, 2023 [–]

No need as s3 objects is your dead letter queue and the system should be designed anyway to coupe with multiple write of same event.

The point is to only use s3 etc in the event of system instability. Not as a primary data transfer means.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact