AWS EC2 Provisioning

Auto Scaling Groups

Auto Scaling Groups are a group of EC2 instances that are scaled out and scaled in response to some events and according to scaling policies, if one is configured. The number of instances is controlled by three variables

Desired Capacity: The number of instances running when Auto Scaling Group is launched

Minimum Capacity: The minimum number of instances running. EC2 Auto Scaling will keep this number of instances regardless of what the scale in event indicated.

Maximum Capacity: The maximum number of instances running. EC2 Auto Scaling will keep this number of instances regardless of what the scale out event indicated.

Launch Templates

The template that AWS EC2 Auto Scaling uses to launch instances. For the instances to be usable the configuration parameters below must be assigned appropriately.

Amazon Machine Image (AMI): The image from which the AWS EC2 instance is launched. The AMI must be AWS ECS optimized, i.e. have the AWS ECS agent.

User Data: The script run when an AWS EC2 is launched. This script must register the name of the AWS ECS cluster to which the instance must belong. In the example below the instance will be resigtered with the cluster acme.

#!/bin/bash
echo ECS_CLUSTER=acme >> /etc/ecs/ecs.config

AWS ECS

Cluster

An ECS cluster is what provides the infrastructure on which the containers are deployed.Every AWS ECS resource is deployed in a cluster.

Tasks

A task is the lowest level of abstraction in AWS ECS, it is where containers are deployed.AWS ECS deploys a task according to parameters defined in a configuration file, called task definition.

The following parameters were important in deploying spaced reps

Container Definitions: An array of JSON documents in which one defines how Docker should run containers, it's effectively a list of dockerfiles.

Execution Role ARN: The Amazon Resource Number(ARN) of the role that Docker daemon and AWS ECS agent use to a deploy a container. Docker uses this role to pull the docker image and run a container while the AWS ECS agent uses it to send logs to CloudWatch.

There is an AWS managed policy, AmazonECSTaskExecutionRolePolicy, with permissions to read from ECR and write to CloudWatch, so it is a matter of attaching that policy to a role and use the ARN of that role.

Network Mode: The network mode that AWS ECS assigns to the network interface of the container.

AWS recommends awsvpc network mode, but in that mode container network interfaces are not assigned public IP addresses so the containers will not have access to the internet without a NAT gateway.

Services

An AWS ECS service is a group of tasks deployed using a given task definition. It is the abstraction that makes the tasks, and in turn the containers, scaleable.

Desired Tasks: The number of tasks to run.

Deployment Type: The service deployment type, either Rolling or Blue/Green.

The Rolling type has two sub configuration values, minimum running tasks % and maximum running tasks %. These values must be set such that there are enough resource for a deployment to be successful. For example a minimum running tasks % of 100 and maximum running tasks % of 150 requires that there are enough resources for 150% of number of tasks to be deployed.

While Blue/Green requires enough resources to deploy double the number of desired tasks to be run.

Network Configuration: The configuration assigned to the network interfaces of the containers launched. This is required only when the network mode in the task definition is set to awsvpc. For the other modes, the containers share the network interface of their host so the network configuration assigned to their hosts' interface applies to them.

Load Balancer: Load balancer to distribute traffic between containers

Capacity Provider: This is the abstraction that ensures the availability of the EC2 instances on which the tasks, i.e. containers, are deployed. It sends the scaling events to the auto scaling groups according to the needs of the AWS ECS service to which it is assigned.

const inlineStyles = (node) => {
  const clone = node.cloneNode();
  const computedStyles = window.getComputedStyle(node);
  Object.keys(computedStyles).forEach((key) => {
    clone.style.setProperty(key, computedStyles.getPropertyValue(key));
  });

  return clone;
}

function traverseDOM(node) {
  if (!node) {
    return;
  }

  const updatedNode = inlineStyles(node);
  for (let child of updatedNode.children) {
    const updatedChild = traverseDOM(child);
    updatedNode.appendChild(updatedChild);
  }

  return updatedNode;
}

The operation performed on each node is inlineStyle. It gets the computed styles of a node and applies them as inline styles.

This is a DOM tree so it is not built in a specific order and each node has many children, so it is not a binary seach tree.Were it a binary search tree, then our traversal order would be pre-order; We perform the operation on the root node first and then traverse on the children from increasing index position, i.e. from left most to right most.

There are two possibilities of uploading a file
1. Send entire file
2. Stream file

Send Entire File

Send entire csv file from browser

  fetch('http://localhost:3000/upload', {
    method: 'POST',
    headers: {
      'Content-Type': 'text/csv' 
    },
    body: file 
  })
  .then(success => console.log(success))
  .catch(error => console.log(error))

The two important points int the server are

1. How to handle request
2. How to stream csv file content as json into pipeline

To get a stream of JSON objects from the csv file, create a stream and pipe that stream into fast-csv.

  const app = require('express')()
  const textBodyParser = require('body-parser').text
  const csv = require('fast-csv')
  const { Readable } = require('stream')
  
  // Handle very large file
  app.use(text({ type: 'text/csv', limit: '500mb' }))
  
  app.post('/upload', (req, res) => {
    const content = Readable.from(req.body)
    content
      .pipe(csv.parse({ headers: true }))
      .on('data', (data) => {
        console.log(data) // Handle JSON object
      })
    res.sendStatus(200)
  })

Stream File

Stream csv file from browser

  //Important that file is sent as FormData
  const data = new FormData()
  data.append('file', file)
  fetch('http://localhost:3000/upload', {
    method: 'POST',
    body: data,
  })
  .then((success) => console.log(success)) 
  .catch((error) => console.log(error))
  

For the server to handle the stream, the HTTP request must have the header Content-Type: multipart/form-data; boundary=aBoundaryString, more info found here.
By sending the file as form data we can avoid having to specify this header.The browser will take care of it.

Use busboy to get the file stream and pipe that to fast-csv to get a stream of JSON objects.

The resulting code

  app.post('/upload', (req, res) => {
    const busboy = new Busboy({ headers: req.headers })
    // Busboy gives us a lot information regarding the file
    busboy.on('file', (__, file) => {
      file.pipe(csv.parse({ headers: true })).on('data', (row) => {
        // Handle data here. Row is a csv row in JSON
        console.log('Row in JSON', row) 
      })
      file.on('end', function () {
        // Handle end case here
        console.log('file ended')
      })
    })
    busboy.on('finish', function () {
      res.writeHead(303, { Connection: 'close', Location: '/' })
      res.end()
    })
    req.pipe(busboy)
  })
  

Aliases

Git aliases allow one to decrease the amount of typing and the need to memorize exact commands.They are defined in .gitconfig file, under the alias section.

[alias]
	c=commit
	lg="git log -1"

In the example above, the aliases c and lg allow one to run git c and git log instead of git commit or git log -1 .

The alias section in my .gitconfig is as follows

[alias]
  a=add
  b=branch
	c=commit
  p=push
  s=status
  # Add a patch
	ap="!git add ${@} -p; git status"
  # Simple Checkout
	co="!git checkout ${@}; git status"
  # Display last commit (HEAD)
	lst="!git log -1 | cat"
  # Search for a branch
	sr="!git branch -a | grep $2"
  # Discard changes
	unco="!git checkout -- ${@}; git status"
  # Unstage changes
	unst="!git reset HEAD -- ${@}; git status"

The ! tells to git to run the command as if in a terminal.

Hooks

Git hooks allow one to tap into the the git lifecycle events.This gives one the ability to automate routine tasks. I use the following hooks

1. pre-commit
2. commit-msg
3. pre-push

Pre-commit

This hook is run before any of the commit related git lifecycle events are fired. I use it to do the linting and type checking.

#!/bin/sh
set -e

changed_files="$(git diff --staged --name-only *.{ts,tsx})"

## Linter on changed files
echo "Running ESLint on changed files"
echo $changed_files | xargs npx eslint

## Typechecker on changed files
echo "Running typechecker on changed files"
echo $changed_files | xargs npx tsc --noEmit

Commit-msg

This hook is run after a commit message has been written and before the commit has been applied.I use it to ensure that the commit message header is prefixed with the issue/ticket id.

#!/bin/sh
set -e

# Read first line from EDIT_COMMITMSG file
read -r msg<$1

# Get current branch name
branch=$(git symbolic-ref --short -q HEAD)

# If branch name is properly named, then extract XXX-1234 prefix.
# It will used to prefix the commit message.
if [[ $branch =~ ^[a-zA-Z]{3}-[0-9]+ ]]; then
  prefix="${BASH_REMATCH[0]}"
else 
  echo "Branch name is not prefixed with XXX-1234."
  exit 1
fi

if [[ $msg =~ ^[a-zA-Z]{3}-[0-9]+ ]]; then
  echo "The commit message properly formatted."
else
  msg="$prefix $msg"
  # Replace the commit message header with the prefixed one
  gsed -i "1s/.*/$msg/" $1
  echo "Prefixed the commit message with $prefix"
fi

Pre-push

This hook is run before one pushes to a remote branch. I use it to run tests, this is specially helpful when pushing to an open PR.

#!/bin/bash

npm test -- --all --watchAll=false --silent