Skip to main content

Quickstart

Not availableCloud AvailableSelf-Managed Community (OSS)Not availableSelf-Managed Enterprise

Airbyte Open Source is a reliable and extensible open source data pipeline.

If you're getting started with Airbyte Cloud, you can skip ahead to moving data by adding your first source.

This quickstart guides you through creating a locally deployed instance of Airbyte in just minutes using abctl (Airbyte Command Line Tool). You'll be able to move data with minimal setup while you're exploring what Airbyte can do!

If you've already set up an Airbyte instance using Docker Compose and want to move to abctl, see the section on migrating from Docker Compose.

tip

When you're ready to put an Airbyte instance into production, you'll want to review our guides on deployment.

For the best experience, we recommend Deploying Airbyte on Kubernetes via Helm.

On a local deployment, Airbyte's default behavior is to store connector secrets in your configured database. These secrets are stored in plain text and are not encrypted. Refer to the Secret Management documentation to set up an external secrets manager.

If setting up an Airbyte server does not fit your use case needs (i.e. you're using Jupyter Notebooks or iterating on an early prototype for your project) you may find the PyAirbyte documentation useful.

Prerequisites​

  • To use abctl, you'll need to have Docker Desktop installed. See Docker's instructions for installation: Mac, Windows, Linux

1: Install abctl​

The easiest method for installing abctl for Mac and Linux users is to use the following command:

curl -LsfS https://get.airbyte.com | bash -

If you would rather install abctl yourself, follow the instructions for your operating system:

We recommend that Mac users use Brew to install the abctl command.

brew tap airbytehq/tap
brew install abctl

With Brew, you can keep abctl up to date easily, by running:

brew upgrade abctl

2: Run Airbyte​

Ensure that Docker Desktop is up and running. Then, with abctl installed, the following command gets Airbyte running:

abctl local install

Your browser may open automatically to the Airbyte Application. If not, access it by visiting http://localhost:8000.

You will be asked to enter your email address and an organization name. Your email address will be used to authenticate to your instance of Airbyte. You will also need a password, which is randomly generated as part of the install command. To get your password run:

abctl local credentials

Which should output something similar to:

{
  "password": "password",
  "client-id": "client_id",
  "client-secret": "client_secret"
}

Use the value in the password field to authenticate to your new Airbyte instance. If you wish to configure authentication follow the documentation on the Authentication Integration page.

As long as your Docker Desktop daemon is running in the background, you can use Airbyte by returning to http://localhost:8000.

If you quit Docker Desktop and want to return to your local Airbyte workspace, just start Docker Desktop again. Once Docker finishes restarting, you'll be able to access Airbyte's local installation as normal.

3: Move Data​

In the Building Connections section, you'll learn how to start moving data. Generally, there are three steps:

1: Set up a Source

2: Set up a Destination

3: Set up a Connection

Customizing your Installation with a Values file​

Optionally, you can use a values.yaml file to customize your installation of Airbyte. Create the values.yaml on your local storage. Then, apply the values you've defined by running the following command and adjusting the path to the values.yaml file as needed:

abctl local install --values ./values.yaml

Here's a list of common customizations.

Migrating from Docker Compose (Optional)​

If you have data that you would like to migrate from an existing docker compose instance follow the steps below:

  1. Make sure that you have stopped the instance running in docker compose, this may require the following command:
docker compose stop
  1. Make sure that you have the latest version of abctl by running the following command:
curl -LsfS https://get.airbyte.com | bash -
  1. Run abctl with the migrate flag set with the following command:
abctl local install --migrate
note

If you're using a version of Airbyte that you've installed with abctl, you can find instructions on upgrading your Airbyte installation here.

Using an EC2 Instance with abctl​

This guide will assume that you are using the Amazon Linux distribution. However. any distribution that supports a docker engine should work with abctl. The launching and connecting to your EC2 Instance is outside the scope of this guide. You can find more information on how to launch and connect to EC2 Instances in the Get started with Amazon EC2 documentation from Amazon.

tip

abctl runs by default on port 8000. You can change the port by passing the --port flag to the local install command. Make sure that the security group that you have configured for the EC2 Instance allows traffic in on the port that you deploy Airbyte on. See the Control traffic to your AWS resources using security groups documentation for more information.

  1. Install the docker engine:
sudo yum install -y docker
  1. Add the ec2-user (or whatever your distros default user) to the docker group:
sudo usermod -a -G docker ec2-user
  1. Start and optionally enable (start on boot) the docker engine:
sudo systemctl start docker
sudo systemctl enable docker
  1. Exit the shell and reconnect to the ec2 instance, an example would look like:
exit
ssh -i ec2-user-key.pem ec2-user@1.2.3.4
  1. Download the latest version of abctl and install it in your path:
curl -LsfS https://get.airbyte.com | bash -
  1. Run the abctl command and install Airbyte:
abctl local install

Editing the Ingress​

By default abctl will install and Nginx Ingress and set the host name to localhost. You will need to edit this to match the host name that you have deployed Airbyte to. To do this you will need to have the kubectl command installed on your EC2 Instance and available on your path.

If you do not already have the CLI tool kubectl installed, please follow these instructions to install.

Then you can run kubectl edit ingress -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig and edit the host key under the spec.rules section of the Ingress definition. The host should match the FQDN name that you are trying to host Airbyte at, for example: airbyte.company.example.

Uninstalling​

If you want to remove Airbyte from your system, consider which of the following two options you would like to use.

1: Run the following command to stop all running containers that abctl has created while preserving any data you've created:

abctl local uninstall

2: If you want to clear the persistent data in addition to stopping containers, run:

abctl local uninstall --persisted

As a last step, to clear out any additional information that abctl may have created, you can run:

rm -rf ~/.airbyte/abctl

Troubleshooting​

There are several channels for community support of local setup and deployment.

GitHub Airbyte Forum's Getting Started FAQ:
Search the questions others have asked or ask a new question of your own in the GitHub forum.

Airbyte Knowledge Base:
While support services are limited to Cloud and Enterprise customers, anyone may search the support team's Help Center.

Community Slack:
Helpful channels for troubleshooting include:

  • #ask-community-for-troubleshooting: Where members of the Airbyte community can ask and answer questions.
  • #ask-ai: For quick answers sourced from documentation and open support channels, you can have a chat with our virtual Airbyte assistant.

Introductory Course:
On Udemy, The Complete Hands-on Introduction to Airbyte is a convenient and hands-on introduction to Airbyte that includes setting up example source and destination configurations. You'll also go on to use it in conjunction with Apache Airflow, Snowflake, dbt, and more.

Bug Reports:
If you find an issue with the abctl command, please report it as a github issue here with the type of 🐛 [abctl] Report an issue with the abctl tool.

Releases:
If you'd like to select which release of abctl to run, you can find the list of releases here.

Was this page helpful?