A Data Warehousing Concept — A data warehouse is an OLAP tool to store and process a massive volume of structured data. One of the objectives is to store historical snapshots of the data coming from the OLTP systems. It means whatever append, update, and delete operations that are performed on the OLTP databases should…

JSON, XML, CSV, Parquet, ORC, and Avro e.t.c — As a data engineer, one of my tasks is to extract and load data. Data is not always extracted and stored in a database. Quite frequently you will have to face use cases where you need to deal with data stored in files. The most common file format in the…

Example: Star Schema, Snowflake Schema, Fact Constellation Schema — Much like an OLTP system(database), an OLAP system(data warehouse) is also required to maintain a schema. A database uses an entity relational model, while a data warehouse uses a dimensional model. In this blog, we will see the three major schemas that are used to model a data warehouse. About Star Schema Each…

Understand technical differences and use cases for Interview purposes — In the Big-Data ecosystem or preparing for a data engineering interview, we often come across these keywords like databases, data lakes, and data warehouses. Since all these tools are used to hold data, this makes us think about what exactly are the key differences. In this blog, we will precisely…

More than df.describe() — While working with data in PySpark, we often do data manipulation in the form of transformations. We receive data from stakeholders or some data pipelines. How do you make sure that the data that is sent to you is correct? …

Write PySpark data frame with specific file names in CSV/Parquet/JSON format — Spark users find it difficult to write files with a name of their choice. The default name that spark uses is the part files. In scenarios where we build a report or metadata file in CSV/JSON format, we want to save it with a specific name. There are some cases…

Data Engineer Interview Questions — This is part six of the data engineer interview series. If you have not already, I highly recommend you to read Part I to build an understanding of my career background. Data Engineer Interview Questions: Part I Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithmmedium.com In this blog, I have discussed the keywords that you might encounter in data engineer interviews. …

Flipkart Interview Experience — This is part five of the data engineer interview series. If you have not already, I highly recommend you to read Part I to build an understanding of my career background. Data Engineer Interview Questions: Part I Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithmmedium.com In this blog, I have discussed a data modeling interview question that was asked me while I was interviewing…