My Study Notes on Amazon Elastic Container Service (Amazon ECS)

Published:
  Lastmod: 2023-01-17
by Ernest Chiang

(Illustration: Brazil’s largest port, Port of Santos, provides container loading and unloading services. Image source: Photo by sergio souza on Unsplash。)

Briefing

(Command/Ctrl + Press the image to enlarge.)

History

From ancient to modern times. Get through all the context.

Terms

Here is a list of nouns that appear on the scene, and the full name of the original text, noun definition and source are noted.

  • Amazon ECS:
    • Amazon Elastic Container Service is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster.
  • Amazon ECS Launch Types:
  • Features of Amazon ECS
    • Containers:
      • To deploy applications on Amazon ECS, your application components must be architected to run in containers. For more information about container technology, see Docker Basics for Amazon ECS.
    • Container Images:
      • Containers are created from a read-only template called an image.
    • Task Definitions:
      • A blueprint for your application. To prepare your application to run on Amazon ECS, you create a task definition. The task definition is a text file, in JSON format, that describes one or more containers, up to a maximum of ten, that form your application.
    • Tasks:
      • A task is the instantiation of a task definition within a cluster.
    • Task Scheduling:
      • The Amazon ECS task scheduler is responsible for placing tasks within your cluster. There are several different scheduling options available. For example, you can define a service that runs and maintains a specified number of tasks simultaneously.
    • Clusters:
      • When you run tasks using Amazon ECS, you place them on a cluster, which is a logical grouping of resources.
    • Amazon ECS Container Instances:
      • When using the EC2 launch type, then your clusters are a group of container instances you manage. An Amazon ECS container instance is an Amazon EC2 instance that is running the Amazon ECS container agent.
    • Container Agent:
      • The container agent runs on each infrastructure resource within an Amazon ECS cluster. It sends information about the resource’s current running tasks and resource utilization to Amazon ECS, and starts and stops tasks whenever it receives a request from Amazon ECS.

Use Cases

  • Suitable: all kinds of industries.
  • Suitable: The initial stage of digital transformation.
    • Encapsulate, isolate, and getting cloud native to get rid of depreciation of fixed assets.
  • Suitable: scenario where auto scaling is done at the granularity of minutes.
  • Not suitable: scenario where auto scaling is done at the granularity of seconds.

(2020) Case Study: rt.live

rt.live, a measure of how COVID-19 is spreading for each state in the United States. Made by Instagram co-founders Mike Krieger and Kevin Systrom after their leaving Facebook.

Mike Krieger published a post on Medium: Automating daily runs for rt.live’s COVID-19 data using Airflow & ECS.

The goals of rt.live:

  1. Ran daily on a schedule
  2. Would re-try steps automatically if transient failures occurred
  3. Could run many US states in parallel, since each states’ ML model run doesn’t depend on other states
  4. Could be manually re-run for a subset of states without rerunning the whole thing, if we needed to make a data correction for just one or two states

The tools used are mainly: Apache Airflow and Amazon’s Elastic Constiner Service (ECS), especially Fargate. By using Fargate, they can build Docker containers as the Airflow workers, and can be scaled to N instances without worrying about creating and maintaining EC2 instances. Also it takes just 30 minutes everyday, that cost is managed.

(Image source: original post)

  • Before starting the calculation, change the ECS service desiredCount to 25 and stand by.
  • Each ECS Task on Fargate is shared and stored in model code of Elastic File System (EFS).
  • Use S3 to store original data from COVIDTracking.
  • Put one Airflow task per state into the queue and let the ECS task (Airflow worker) execute it. The calculation result is also stored in S3.
  • After the calculation of the day is completed, change the ECS service desiredCount back to 0.

Other extended reference materials, you can create similar pipelines: How to deploy Apache Airflow with Celery on AWS and Lessons learned deploying Apache Airflow to ECS.

Limits

Adjustable

Service quotaDescriptionDefault
ClustersThe maximum number of clusters in this account in the current Region.10,000
Container instances per clusterThe maximum number of container instances per cluster.2,000
Services per clusterThe maximum number of services per cluster.5,000
Tasks per serviceThe maximum number of tasks per service (the desired count).5,000
Tasks using the EC2 launch type per service (the desired count)The maximum number of tasks using the EC2 launch type per service (the desired count). This limit applies to both standalone tasks and tasks launched as part of a service.1,000
Tasks using the Fargate launch type or the FARGATE capacity provider, per Region, per accountThe maximum number of tasks using the Fargate launch type or the FARGATE capacity provider, per Region. This limit applies to both standalone tasks and tasks launched as part of a service.100
Fargate Spot tasks, per Region, per accountThe maximum number of tasks using the FARGATE_SPOT capacity provider, per Region.250
Public IP addresses for tasks using the Fargate launch typeThe maximum number of public IP addresses used by tasks using the Fargate launch type, per Region.100

Non-Adjustable

Service quotaDescriptionDefault
Tasks launched (count) per run-taskThe maximum number of tasks that can be launched per RunTask API action.10
Container instances per start-taskThe maximum number of container instances specified in a StartTask API action.10
Revisions per task definition familyThe maximum number of revisions per task definition family. Deregistering a task definition revision does not exclude it from being included in this limit.1,000,000
Task definition size limitThe maximum size, in KiB, of a task definition.32
Task definition max containersThe maximum number of containers definitions within a a task definition.10
Subnets specified in an awsvpcConfigurationThe maximum number of subnets specified within an awsvpcConfiguration.16
Security groups specified in an awsvpcConfigurationThe maximum number of security groups specified within an awsvpcConfiguration.5
Target groups per serviceThe maximum number of target groups per service, if using an Application Load Balancer or a Network Load Balancer.5
Classic Load Balancers per serviceThe maximum number of Classic Load Balancers per service.1
Tags per resourceThe maximum number of tags per resource. This applies to tasks, services, task definitions, clusters, and container instances.50

Pricing

Please refer to the latest official documents. This is just shorthand.

  • According to Amazon ECS Launch Types, there are two categories.
    • Fargate Launch Type Model
      • Charge according to the Fargate resources used. Refer to AWS Fargate pricing page. There is no additional charge for Amazon ECS.
    • EC2 Launch Type Model
      • Charge according to the EC2 resources used. Refer to Amazon EC2 pricing page. There is no additional charge for Amazon ECS.
  • Amazon ECS on AWS Outposts Cf. above EC2 Launch Type Model.

Deep Dive

The general direction is a path, but beware of pits on the ground.

Reference

Awesome Lists

Getting Started

Examples

Articles & Talks

Comparison

Loading comments…