Andrii Soldatenko

Senior Data Engineer at Toptal, Ukraine

Python developer in the day, Go developer (gopher) under the hood. Big fan of full-text search and graph databases.

Top 3% of Freelance Talent (Toptal is an exclusive network of the top freelance software developers, designers, and finance experts in the world).

Contributed in different python/go open source projects:
– pyhelm, aiohttp-swagger, mezzanine
– chalice, requests, aiohttp tutorial
– sendgrid-python and sendgrid-django
– OpenAPI v3 specification, fix Go docs

Speaker at PyCaribbean, PyCon Italia 2017, EuroPython 2016, PyCon Ukraine 2014, PyCon Belarus 2015-2018 PyCon Russia 2015, 2016.

Blogger at https://asoldatenko.com/.

Speaker's activity

The art of data engineering

December 7th

11:05

Talk

Russian

As the the data space has increased, data engineering has emerged as a separate and related role that works together with data scientists. Usually data scientists focus on finding new insights from a data set, while data engineers are concerned with the production readiness of that data.

In this talk I’ll show you how to gather and collect the huge amount of data, stores it, does batch processing or real-time processing on it, and how to build data pipeline using Airflow for processing billion of records per table.

Also we will discuss what is big data, and why it’s important to be able to process it so quick.