Kulasangar Gowrisangar

Pinned

Creating and attaching an AWS IAM role, with a policy to an EC2 instance using Terraform scripts

What is Terraform? — This is an infrastructure as a code, which is equivalent to the AWS CloudFormation, that allows the user to create, update, and version any of the Amazon Web Services (AWS) infrastructure. Why Terraform? Terraform utilizes the cloud provider APIs (Application programming interfaces) to provision infrastructure, hence there’re no authentication techniques after, what…

AWS

4 min read

AWS

4 min read

Published in Level Up Coding

·Dec 26, 2022

Using Python Faker library to generate dummy data

What is dummy data? This type of data is typically used in Software Development, Unit Testing, ML model training, Data Analysis etc to prevent from using sensitive data for the above use cases. Also while generating random data, you as a developer can make sure to cross-check all your validations when it comes to…

Faker

3 min read

Faker

3 min read

Published in Geek Culture

·Jan 17, 2022

Streaming Twitter Feed into Elasticsearch and MySQL using Apache NiFi

In this blog post I’ll be explaining on how we can easily extract Twitter feed, stream it into Elasticsearch in real time, analyze and visualize the data in Kibana and also inserting the feed into MySQL database in near real time, by using Apache NiFi. What is Apache NiFi? Apache…

Apache Nifi

7 min read

Apache Nifi

7 min read

Published in Geek Culture

·Jul 8, 2021

What is SchemaCrawler and Why would you need it?

In this blog post, I’m going to explain how you can use Schema Crawler tool in order to bring in table details from a given schema. …

Schemacrawler

5 min read

Schemacrawler

5 min read

Published in Level Up Coding

·Jan 23, 2021

What is AWS Data Pipeline and how it can be used?

I was listing down some of the AWS web services that I haven’t personally touched based on, and that’s when I stumbled upon AWS Data Pipeline. …

Aws Data Pipeline

7 min read

Aws Data Pipeline

7 min read

Published in Level Up Coding

·Oct 29, 2020

Use Presigned URL to upload files into AWS S3

In this blog post, I will be walking through the steps as to how we can utilize the presigned url feature to upload files into AWS S3. Serverless will be used to spin up the necessary AWS resources for this post. Why do we need a Presigned URL in the first place? Presigned URL can be used in an instance such…

AWS

4 min read

Use Presigned URL to upload files into AWS S3

AWS

4 min read

Nov 5, 2018

Create an EMR cluster and submit a job using Boto3

Steps: — Creating a job to submit as a step to the EMR cluster. Copy the executable jar file of the job we are going to execute, into a bucket in AWS S3. Creating an AWS EMR cluster and adding the step details such as the location of the jar file, arguments…

AWS

3 min read

Create an EMR cluster and submit a job using Boto3

AWS

3 min read

Sep 28, 2018

Communicate between two different applications which reside in two different Docker containers (Vue.js, PHP + MySQL)

What is Docker? — Docker is a mechanism that is created to assist both system administrators and the developers , making it a component of many DevOps tool chains. From the developers’ perspective, they can focus on writing code without having any concerns about the system, that it will eventually be running on. …

Docker

4 min read

Docker

4 min read

Aug 7, 2018

Creating a Spark job using Pyspark and executing it in AWS EMR

What is Spark? Spark is considered as one of the data processing engine which is preferable, for usage in a vast range of situations. Data Scientists and application developers integrate Spark into their own implementations in order to transform, analyze and query data at a larger scale. …

Pyspark

4 min read

Pyspark

4 min read

Jul 21, 2018

DATA LAKE REFERENCE ARCHITECTURE

INTRODUCTION Data lake is a single platform which is made up of, a combination of data governance, analytics and storage. It’s a secure, durable and centralized cloud-based storage platform that lets you to ingest and store, structured and unstructured data. It also allows us to make necessary transformations on the raw…

Big Data

4 min read

Big Data

4 min read