u/ASlowReader

Been working for several years at a small company as a “Python Developer” working on and maintaining scrapers using Selenium, Scrapy, BeautifulSoup, Pandas, etc. I’m familiar with CSVs, JSON, consumption of APIs and know basic SQL.

We have a process where data is regularly scraped from the web on a schedule, formatted in a way we want, then gets saved as a CSV, which then gets WinSCPed to an SFTP server. All done using Python or Batch scripts.

I was wondering if this type of work classes as a Data Engineer. Does what I’ve described above qualify as a data pipeline or is there more to it?