Projects/zed-lake

Create a Remote Zed Lake

Checklist of resources

• An SSH key pair
• A cloud-config template
• An AWS Account and permissions to create resources
• A VPC
• A Security Group
• An AMI (Amazon Linux should fit in the free tier too)
• An EC2 instance in the free tier (think t2.micro or t3.small)
  • If this is for production, right size your EC2 instance

Prep Work

Create the SSH key pair. (ED25519 is used due to shortened key length and stronger security but RSA is still viable)

ssh-keygen -t ed25519 -f ~/.ssh/zed -C "zed lake"

Add a password if you desire, then add your key to the ssh-agent.

ssh-add .ssh/zed

And enter your password if you opted for a password.
You should have some output like:

Identity added: /home/your-username/.ssh/zed (zed lake)

Create the cloud-config template

We'll go over the template in pieces, then look at the complete file. This template is based on RHEL-based Linux distributions, but can easily be tailored to a different distribution

The users section

The user section is where you can add all user accounts and the relevant configuration of those users:

#cloud-config
users:
  - name: zed
    lock_passwd: false
    passwd: some-passwd-hashed
    ssh_authorized_keys:
      - your-ssh-PUBLIC-KEY-here
    shell: /bin/bash
    groups: wheel
    

Here we have:

• The #cloud-config top line let's cloud-init. That line is necessary for cloud-init to process the file.
• name is the username
• lock_passwd: false makes sure the user we're creating isn't locked. For this to remain true, the user needs a...
• passwd a password for said user. The openssl passwd -6 command is great for allowing you to create a salted password
• ssh_authorized_keys: add the public key of your zed user here
• shell is the default login shell for the zed user
• groups is the groups the zed user belongs to

Optionally, the line sudo: ['ALL=(ALL) NOPASSWD: ALL'] can be added.
This will allow the zed user to issue commands as root without prompting for a password. Care should be taken if this option is added to the config.

The write files section

We can leverage cloud-init to write a couple files:

write_files:
  - path: /etc/ssh/sshd_config
    content: |
         Protocol 2
         HostKey /etc/ssh/ssh_host_rsa_key
         HostKey /etc/ssh/ssh_host_dsa_key
         HostKey /etc/ssh/ssh_host_ecdsa_key
         HostKey /etc/ssh/ssh_host_ed25519_key         
         SyslogFacility AUTH
         LogLevel INFO
         LoginGraceTime 120
         PasswordAuthentication no
         PermitRootLogin no
         StrictModes yes
         PubkeyAuthentication yes
         IgnoreRhosts yes
         HostbasedAuthentication no
         PermitEmptyPasswords no
         ChallengeResponseAuthentication no
         PrintLastLog yes
         TCPKeepAlive yes
         AcceptEnv LANG LC_*         
         UsePAM yes
  - path: /usr/lib/systemd/system/zed-lake.service
    content: |
         [Unit]
         Description=Zed Lake Service
         After=network.target
         
         [Service]
         ExecStart=/opt/Zui/resources/app.asar.unpacked/zdeps/zed serve -l :9867 -lake /home/zed/.config/Zui/lake -log.level=info -log.filemode=rotate -log.path=/home/zed/.config/Zui/logs/zlake.log
         Restart=always
         User=zed
         
         [Install]
         WantedBy=multi-user.target

This section tells cloud-init to do the following:

• We're writing a new sshd_config to define some rules, mostly to enforce public key authentication only and no root login
• We're creating a systemd service for our zed serve command to run automatically. At a high level we're:
  • Telling the zed serve binary to listen on port 9867 on all interfaces:
    • We'd usually define an IP here but we're not sure what IP AWS will assign us
    • This can be changed once we know the IP
  • Defining the lake
  • Defining the log level
  • Defining log rotation
  • Defining the log path

The packages section

We can have packages installed cloud-init update the OS repositories and install packages during provisioning:

package_update: true
package_upgrade: true
packages:
  - plocate
  - unzip
  - tar
  - vim
  - firewalld
  

Here we're updating package repositories and any packages that are on the system already, chiefly firewalld.

The run runcmd section

This section will all define commands we want to run during the cloud-init process. This will round out the template and complete the setup.

runcmd:
  - systemctl enable --now firewalld
  - firewall-cmd --permanent --add-port 9867/tcp
  - firewall-cmd --reload
  - dnf install -y https://github.com/brimdata/zui/releases/download/v1.18.0/Zui-1.18.0.x86_64.rpm
  - mkdir -p /home/zed/.config/Zui/lake /home/zed/.config/Zui/plugins/brimcap/storage/root /home/zed/.config/Zui/logs
  - chown zed:zed -R /home/zed/.config
  - find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \;
  - /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater
  - systemctl daemon-reload
  - systemctl enable --now zed-lake.service

Here cloud-init is:

• Enabling the firewalld service (it usually has its preset as enabled but this will ensure that)
• Adding TCP port 9867
• Reloading firewalld
• Installing the Zui app:
  • The Zui app has all the binaries we need for a zed lake, however the zed binary can be installed alone
  • Some further configuration for packet analysis
• Creating some directories for Zed. Some of these you saw in the systemd service we created earlier in the template
• Setting ownership back to our zed user. The actions taken by cloud-init are performed by root, so we'll need to restore ownership or our service will fail to start
• Configuration to enable packet analysis
• Reloading all daemons on the system, as we created a new one for the zed-lake.service
• Enabling the zed-lake.service to auto-start

The template in its entirety should look like:

#cloud-config
users:
  - name: zed
    lock_passwd: false
    passwd: some-passwd-hashed
    ssh_authorized_keys:
      - your-ssh-PUBLIC-KEY-here
    shell: /bin/bash
    groups: wheel
write_files:
  - path: /etc/ssh/sshd_config
    content: |
         Protocol 2
         HostKey /etc/ssh/ssh_host_rsa_key
         HostKey /etc/ssh/ssh_host_dsa_key
         HostKey /etc/ssh/ssh_host_ecdsa_key
         HostKey /etc/ssh/ssh_host_ed25519_key         
         SyslogFacility AUTH
         LogLevel INFO
         LoginGraceTime 120
         PasswordAuthentication no
         PermitRootLogin no
         StrictModes yes
         PubkeyAuthentication yes
         IgnoreRhosts yes
         HostbasedAuthentication no
         PermitEmptyPasswords no
         ChallengeResponseAuthentication no
         PrintLastLog yes
         TCPKeepAlive yes
         AcceptEnv LANG LC_*         
         UsePAM yes
  - path: /usr/lib/systemd/system/zed-lake.service
    content: |
         [Unit]
         Description=Zed Lake Service
         After=network.target
         
         [Service]
         ExecStart=/opt/Zui/resources/app.asar.unpacked/zdeps/zed serve -l :9867 -lake /home/zed/.config/Zui/lake -log.level=info -log.filemode=rotate -log.path=/home/zed/.config/Zui/logs/zlake.log
         Restart=always
         User=zed
         
         [Install]
         WantedBy=multi-user.target
package_update: true
package_upgrade: true
packages:
  - plocate
  - unzip
  - tar
  - vim
  - firewalld
runcmd:
  - systemctl enable --now firewalld
  - firewall-cmd --permanent --add-port 9867/tcp
  - firewall-cmd --reload
  - dnf install -y https://github.com/brimdata/zui/releases/download/v1.18.0/Zui-1.18.0.x86_64.rpm
  - mkdir -p /home/zed/.config/Zui/lake /home/zed/.config/Zui/plugins/brimcap/storage/root /home/zed/.config/Zui/logs
  - chown zed:zed -R /home/zed/.config
  - find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \;
  - /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater
  - systemctl daemon-reload
  - systemctl enable --now zed-lake.service

With the template created, we can move to AWS.

The Cloud Side

AWS Tasks

• Create a VPC
• Create a security group and security group rules
• Create a subnet
• Create an Internet Gateway (IGW)
• Create a Route table and route to allow internet access for the subnet
• Create the EC2 instance

Creating the VPC

Probably the easiest part of our setup.

aws ec2 create-vpc --cidr-block 10.10.0.0/16 --tag-specifications ResourceType="vpc",Tags=['{Key=Name,Value=zed_vpc}']

Record the VpcId from the output, we'll need it again shortly.

Now we need to create a subnet within our VPC

aws ec2 create-subnet \
    --vpc-id your-vpc-id \
    --cidr-block your-cidr-block \
    --tag-specifications ResourceType=subnet,Tags=[{'Key=Name,Value=zed_subnet}']
    

Record the SubnetId. That'll be used shortly as well.

Note: While tagging isn't necessary it helps to have descriptions of resources. So tag responsibly. Another Note: If you're using a bash shell, make sure to place quotes around your tags. It will throw an error stating its looking for a string and you provided a 'dict'.

The subnet where the EC2 instance will reside will need internet access in order for the Zui app on the local endpoint to be able to access the Zed lake on the EC2:

Provide the subnet internet access

• aws ec2 create-internet-gateway to create the IGW, record the InternetGatewayId
• aws ec2 attach-internet-gateway --vpc-id your-vpc-id --internet-gateway-id your-igw-id to attach the gateway to our VPC
• aws ec2 create-route-table --vpc-id your-vpc-id to create a route table associated with your VPC. Record your RouteTableId
• aws ec2 create-route --route-table-id your-route-table-id --destination-cidr-block 0.0.0.0/0 --gateway-id your-igw-id to create the specified route:
  • Note: The IP/Network specified in the security group rules (x.x.x.x/32) here as well as an added security layer alongside the security group rules that are created later
• aws ec2 associate-route-table --subnet-id your-subnet-id --route-table-id your-route-table-id to associate the route with the subnet we created above

Creating the security group

Now that there's a by way of the VPC, a security group can be created for the VPC. A security group acts like a firewall, where ingress and egress rules can be added to accept or deny network traffic.

aws ec2 create-security-group --group-name ZedSecGroup --description "Zed Lake Security Group" --vpc-id <your-vpc-id>

Record the GroupId.

Creating the Security Group rules

Now to create the security group rules (descriptions are optional)

aws ec2 authorize-security-group-ingress \
    --group-id your-security-group-id \
    --ip-permissions 'IpProtocol=tcp,FromPort=22,ToPort=22,IpRanges=[{CidrIp=x.x.x.x/32,Description="Allow SSH access from my IP"}]'

aws ec2 authorize-security-group-ingress \
    --group-id your-security-group-id \
    --ip-permissions 'IpProtocol=tcp,FromPort=9867,ToPort=9867,IpRanges=[{CidrIp=x.x.x.x/32,Description="Allow Zed lake access from my IP"}]'
    

Creating the EC2 instance

EC2 Instance creation

• Choose an AMI (The OS)
• Choose an Instance Type(The machine architecture)

aws ec2 run-instances \
    --image-id ami-03c4f11b50838ab5d \
    --instance-type t3.small \
    --user-data file://zed-lake.yaml \
    --metadata-options "HttpEndpoint=enabled,HttpTokens=required" \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=zed-lake}]' \
    --network-interfaces \
        '{"DeviceIndex":0,"SubnetId":"your-subnet-id","Groups":["sg-your-security-groupid"],"AssociatePublicIpAddress":true}'
    

Once the machine ready, test the SSH rule by attempting to access the instance

ssh zed@<ec2-instance-ip-address>

Also check the status of the zed-lake.service

systemctl status zed-lake.service

Which should show running and enabled

And see if the EC2 is listening of the Zed lake port (9867/tcp)

ss -lt

Connecting to the Zed Lake

Now that the actual work is over, we can open up Zui on our local machine and connect to the lake we just created.

And add some data.

And select Query Pool

The Zed Lake is ready for data analysis. Consult the docs on how to formulate queries and transform the data. It isn't quiet SQL, but if you're familiar with SQL it shouldn't take long to get comfortable with it.