Experiments with ECS: 1 - Deploying 1 microservice in ECS

 

NOTE - 

a. Should have ideally kept the RDS database in a Private Subnet

b. Could also have kept the ECS cluster in Private Subnet (Ref URL - https://repost.aws/knowledge-center/ecs-fargate-tasks-private-subnet)

Step 1 - Create a flask application that talks to a locally installed Postgres database

Ref URL - REST API Design Principles: https://www.freecodecamp.org/news/rest-api-best-practices-rest-endpoint-design-examples/

1.1 Create a Flask application

from flask import Flask

app = Flask(__name__)

@app.route(‘/')

def hello_world():

    return’ Hello World’

if __name__ == ‘__main__’:

    app.run(debug=True)

Tested @ http://127.0.0.1:5000/

1.2 Connect the application to Postgres

from flask import Flask, abort, jsonify

import psycopg2

app = Flask(__name__)

myconn = psycopg2.connect(database = "DataWarehouseX", user = "postgres", password = "xxxxx", host = "localhost", port = "5432")

mycursor = myconn.cursor()

@app.route("/api/1.0/products")

def products_view():

    try:

        global mycursor

        mycursor.execute("SELECT * FROM core.dim_product")

        db=[]

        for x in mycursor:

            db.append(x)        

        return jsonify(db)

    except IndexError:

        abort(404)

        

@app.route("/api/1.0/products/<id>")

def product_view(id):

    try:

        global mycursor

        cmd = "SELECT * FROM core.dim_product where product_id = " + "'"+id+"'"

        mycursor.execute(cmd)

        db=[]

        for x in mycursor:

            db.append(x)        

        return jsonify(db)

    except IndexError:

        abort(404)

if __name__ == '__main__':

    app.run(debug=True)

API in action -

Step 2 - Dockerize the application and run containers locally

Ref URL - https://www.freecodecamp.org/news/how-to-dockerize-a-flask-app/

app.py -

#To connect to the localhost postgresDB, switched to 'host.docker.internal'

from flask import Flask, abort, jsonify

import psycopg2

app = Flask(__name__)

myconn = psycopg2.connect(database = "DataWarehouseX", user = "postgres", password = "xxxx", host = "host.docker.internal", port = "5432")

mycursor = myconn.cursor()

@app.route("/")

def hello_world():

    return 'Hello from docker!'

@app.route("/api/1.0/products")

def products_view():

    try:

        global mycursor

        mycursor.execute("SELECT * FROM core.dim_product")

        db=[]

        for x in mycursor:

            db.append(x)        

        return jsonify(db)

    except IndexError:

        abort(404)

        

@app.route("/api/1.0/products/<id>")

def product_view(id):

    try:

        global mycursor

        cmd = "SELECT * FROM core.dim_product where product_id = " + "'"+id+"'"

        mycursor.execute(cmd)

        db=[]

        for x in mycursor:

            db.append(x)        

        return jsonify(db)

    except IndexError:

        abort(404)

if __name__ == '__main__':

    app.run(debug=True)

Dockerfile -

FROM python:3.8-slim-buster

WORKDIR /python-docker

COPY requirements.txt requirements.txt

RUN pip3 install -r requirements.txt

COPY . .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]

Requirements.txt

#Had issues using the flask version I was using in Spyder. After checking the below page, decide to remove the flask version from requirements.txt

#https://stackoverflow.com/questions/71718167/importerror-cannot-import-name-escape-from-jinja2

#Had issues building docker image with psycopg2.  Hence, switched to psycopg2-binary

flask

psycopg2-binary

Step 3 - Create a RDS postgres database, load data into it, switch to using it in your rest API application

Ref URL - https://sakyasumedh.medium.com/deploy-backend-application-to-aws-ecs-with-application-load-balancer-step-by-step-guide-part-1-91935ae93c51

Had to create a Public database so that can connect to it from my LVDI and load data. Plus also connect to it from my dockerized rest API application on my local machine

Data migration -

Ref URL - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html

Exported data from local database table into a CSV file

Imported data from CSV file into RDS database table

Connection from container to WWW works out of the box

Hence, only had to make the following change to the app.py file -

myconn = psycopg2.connect(database = "DataWarehouseX", user = "postgres", password = "xxxx", host = "rest-ecs-db.c6nvu354y8s3.us-east-1.rds.amazonaws.com", port = "5432")

Also had to fix the inbound rule for the RDS database instance to allow inbound connection from anywhere on port 5432

Step 4 - Create an ECR repo and push images to that repo

Ref URL - https://sakyasumedh.medium.com/deploy-backend-application-to-aws-ecs-with-application-load-balancer-step-by-step-guide-part-2-e81d4daf0a55

Created an ECR repo

Ran AWS configure in Visual Studio terminal where the image file exists

Then followed the push commands specified on the ECR console

Step 5 - Setup ECS cluster and deploy application to ECS

Ref URL - https://sakyasumedh.medium.com/deploy-backend-application-to-aws-ecs-with-application-load-balancer-step-by-step-guide-part-3-b8125ca27177

Had to expose port 5000

Step 6 - Add a load balancer

Ref URL - https://sakyasumedh.medium.com/setup-application-load-balancer-and-point-to-ecs-deploy-to-aws-ecs-fargate-with-load-balancer-4b5f6785e8f

Had to expose port 5000 everywhere (in ALB listened, in Target Group etc.)

Step 7 - Add a custom domain name

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html

Had to register domain. Will automatically create a public hosted zone.

Creating a CNAME record like 

api.anandmusings.link pointing to the alb - rest-ecs-rds-alb-445658067.us-east-1.elb.amazonaws.com worked

Alias record wasn't working for some reason. Deleted a few times and created afresh. And that too started working 

Git Tutorial

A chance conversation with Aishwarya drove a desire to finally further clarify my understanding of git. Hope you find this information useful.

Git is a version control and collaboration tool

- Version control - To allow easy rollback to a previous version of code <you can also achieve this by - creating a copy of the code folder, adding metadata showing what has changed between the 2 copies. Btw, a git repository is also just a folder where all your project files and the related metadata resides>

- Collaboration - To enable collaboration between multiple developers on a single project <you can also achieve this by say zipping the folder and sending it to someone. While merging, we will have to ask what all files have been modified and then manually copy paste the modified code to create a single golden copy of the code>

The question is how to scale the stuff listed within <> -

- Version control - Keeping multiple copies of entire folder would imply we might soon run out of disk space

- Collaboration - 1000s of developers across the globe, who might not even know each other, would need to collaborate to create a single golden copy of the code

That is where git comes in.

Reference URL - https://www.crio.do/blog/what-is-git

Reference URL - https://www.nobledesktop.com/learn/git/what-is-git

Reference URL -

https://www.quora.com/Does-git-make-a-copy-of-all-my-files-each-time-I-make-a-commit

What is the typical git workflow?

Sequence of steps to follow for various scenarios:

Scenario 1 - You want to make changes to an open source project 

Solution -

[IN GITHUB] Step 1 - Fork: Create a copy of the code from the open source code repo A to your own repo B so that you can make changes to it 

[IN GIT BASH ] Step 2 - cd to the laptop location you want to place the code in

[IN GIT BASH ] Step 3 - Clone: Download code from repo B to your local machine

git clone <repo_url>

[In GIT BASH - ONE TIME] Step 4 - Config: Config git

git config --global user.name <your_user_name>

git config --global user.email <your_email>

[In GIT BASH] Step 5 - Code changes - Make code changes 

[In GIT BASH - OPTIONAL] Step 6 - Status - Check which files have been newly created or modified in the working directory 

git status

[In GIT BASH] Step 7 - Add - Start tracking files

git add .

Can also add files individually

git add <file1> <file2> 

Use space to separate file names

NOTE - This tells Git to take a snapshot of the contents of all files under the current directory and add them to the staging area / index (think of it as the backstage of a theatre). Why can't we commit the files directly? Let us say you are working on two files, but only one of them is ready to commit. You don't want to be forced to commit both the files, just the one that is ready

[In GIT BASH - OPTIONAL] Step 8 - Status - Check if the newly created / modified files from step 8 are getting tracked now

git status

[In GIT BASH] Step 9 - Commit - Commit changes to local repo

git commit -m 'Commit message summarising the changes made'

[In GIT BASH] Step 10 - Push - Push changes to remote repo

git push -u origin main

NOTE - You will be promoted to enter your GitHub credentials in the browser 

NOTE - Earlier main was referred to as master but later that was changed to main as master is a bad word just like slave

[IN GITHUB] Step 11 - Pull Request - Create a Pull Request from your repo B back to repo A to merge changes there. 

Sometimes when we merge two branches (from two different repos, as is scenario here or the same repo, as in the next scenario) and two developers have worked on the same part of the file, you will get a merge conflict. Git will show you both sets of changes and let you decide which one do you want to keep.

---

Scenario 2 - You want to make changes to your own team's codebase (FIRST TIME)

[IN GIT BASH ] Step 1 - cd to the laptop location you want to place the code in

[IN GIT BASH ] Step 2 - Clone: Download code from repo to your local machine

git clone <repo_url>

[In GIT BASH - ONE TIME] Step 3 - Config: Config git

git config -Global user.name <your_user_name>

git config -Global user.email <your_password>

NEW! [In GIT BASH] Step 4 - Branch - Create and move to a new branch

git branch branch_name

git checkout branch_name

NOTE - You can also do this in only one command using 

git checkout -b branch_name

[In GIT BASH] Step 5 - Code changes - Make code changes 

[In GIT BASH - OPTIONAL] Step 6 - Status - Check which files have been newly created or modified in the working directory 

git status

[In GIT BASH] Step 7 - Add - Start tracking files

git add .

[In GIT BASH - OPTIONAL] Step 8 - Status - Check if the newly created / modified files from step 8 are getting tracked now

git status

[In GIT BASH] Step 9 - Commit - Commit changes to local repo

git commit -m 'Commit message summarising the changes made'

MODIFIED! [In GIT BASH] Step 10 - Push - Push changes to remote repo

git push -u origin branch_name

MODIFIED! [IN GITHUB] Step 11 - Pull Request - Create a Pull Request from your branch to 'develop' branch to merge changes there

NOTE - Why to the develop branch? We usually setup a CICD pipeline to continually deploy changes from dev branch to the dev environment. When the changes have been tested in dev, we can raise a Pull Request from dev branch to main branch to merge changes to main and use a CICD pipeline to automatically deploy them to prod environment.

---

Scenario 3 - You want to make changes to your own team's codebase (ONGOING)

[IN GIT BASH ] Step 1 - cd to the laptop location you had earlier cloned the repo to

NEW! [IN GIT BASH ] Step 2 - Pull: Do a git pull to download the latest code from the remote repo

git pull

[In GIT BASH - ONE TIME] Step 3 - Config: Config git

git config -Global user.name <your_user_name>

git config -Global user.email <your_password>

[In GIT BASH] Step 4 - Branch - Create and move to a new branch

git branch branch_name

git checkout branch_name

NOTE - You can also do this in only one command using 

git checkout -b branch_name

[In GIT BASH] Step 5 - Code changes - Make code changes 

[In GIT BASH - OPTIONAL] Step 6 - Status - Check which files have been newly created or modified in the working directory 

git status

[In GIT BASH] Step 7 - Add - Start tracking files

git add .

[In GIT BASH - OPTIONAL] Step 8 - Status - Check if the newly created / modified files from step 8 are getting tracked now

git status

[In GIT BASH] Step 9 - Commit - Commit changes to local repo

git commit -m 'Commit message summarising the changes made'

[In GIT BASH] Step 10 - Push - Push changes to remote repo

git push -u origin branch_name

[IN GITHUB] Step 11 - Pull Request - Create a Pull Request from your branch to 'develop' branch to merge changes there

---

Scenario 4 - You want to push changes to a new GitHub repo

NEW! [IN GITHUB] Step 1 - Create repo: Ceate a new repo in GitHub 

[IN GIT BASH ] Step 2 - cd to the laptop folder your code exists 

NEW! [In GIT BASH] Step 3 - Init: Initialize a new git repo

git init

NOTE - This will create a new hidden folder on the directory called .git

[In GIT BASH - ONE TIME] Step 4 - Config: Config git

git config -Global user.name <your_user_name>

git config -Global user.email <your_password>

[In GIT BASH - OPTIONAL] Step 5 - Status - Check which files have been newly created or modified in the working directory 

git status

[In GIT BASH] Step 6 - Add - Start tracking files

git add .

Can also add files individually

git add <file1> <file2> 

Use space to separate file names

[In GIT BASH - OPTIONAL] Step 7 - Status - Check if the newly created / modified files from step 8 are getting tracked now

git status

[In GIT BASH] Step 8 - Commit - Commit changes to local repo

git commit -m 'Commit message summarising the changes made'

NEW! [In GIT BASH - ONE TIME] Step 9 - Set Origin - Set remote repo as origin

git remote add origin <remote repo url>

[In GIT BASH] Step 10 - Push - Push changes to remote repo

git push -u origin main

NOTE - You will be promoted to enter your GitHub credentials in the browser 

This attachment summarizes the difference in steps one needs to follow for each scenario